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FOREWORD 

0 

The pu3Tose of this manual is to provide vocational educators with evalua- 
tion elehients and test€*d. models which can assist them in designing evaluation 
systems . Through the selection of appropriate evaluation mechanisms » voca* 
tional educators will be better able to determine the efficacy of current pro- 
grams. Included is a practical bow-to-do-it guide with numerous models vrtiicb 
can be adapted or modified- to fit any situation. 

Our emphasis on the use of criteria as an evaluation tool should aid in 
the development of local guidelines. The selected models focus on systematic 
vocational evaluation at all levels. The appendices illustrate several con- 
temporary evaluation forms by which to judge both curriculum objectives and 
instructional processes. ^ 
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These models are presented to busy vocational educators so they may adapt 
those which are appropriate to their own local situations. 
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CHAPTER ,1 

^ .CRHEklA FOR VOCATIONAL PROGRAM EVALUATION 

' ' ' - 

INTRODUCTION 

Evaluation of any vocational prqgram or project is often a cornplex yet 
subjective activity* Often the evaluator or evaluating team is ^inexperienced, 
in 4^aluation and roay relj^ on intuitive or. inipressioi^stic approaches**** despite 
books, monographs and articles written about the subject. Oft^n there is no 
specific program evaluation model "Which is appropriate to a local situation* 
Too, there even seems to be a resistance to the concept of evaluation since 
the term usually denotes reward or punishment. The writers of this *guide 
recognize these problems and attitudes. We also recognize the key role which 
an evaluator plays in the judging^ of staff coiqpetencer and program development* 
IhuSf this monograph is addressed ^to those who are responsible for vocational 
evaluation so that they may become knowledgeable about the assumptiions which 
undergir^J any evaluation system. Included are selected evaluation models and 

sets . of evaluativ-e^criteria. All of these elements caff then be used collec* 

% - - - ■ " 

tively to build models appropriate to any individual situation^_ 
Some Tenets Associated Wit.h Evaluatioh • * 

Evaluation, per se, and, more specifically, evaluation of vocational pro*' 
grams must be viewed as ,a process rather than as an all' encompassing^ proce- 
dure. If one seeks 'THE" evaluation procedure, thea that person soon dis*' 
• covers that no single evaluation procedure will ever fit the many facets of 



We wish to acknowledge the work of Betty A- bhrt which is represented, 
in part,* in this chapter. , 



vocational education. 

* . - , . ' ^ V 

«To be most ^ successful I evaluation cannot be v;ieved simply as "manage- 
ment tool"*but as\ a means to an end—the improvement of programs or indivi^iual 
performances* This assumption should help overcome teacher and administrator^ 

resistance^ sihce evaluation then becomes both useful to those involved and 

. # _ ^ ' t ^ 

informative ^to decision-^makers. Evaluation also becomes flexible and is 
treated as an ongoing activity aimed at identifying* both program strengths and 
weaKnesses*9 Such a perspective also .iJSes evaluation as a process for con- 
tinuous redirection of terminal outcomes. The choice obviously involves 
program directors^ instructors^ administra|:ors and iocal4citi;;ens. 

To be sure, specific assessment and measurement procedures are ijsed to 
evaluate a.^speclfic training program. But, the essence of such activities is 
to provide* meaningful feedback. ' * ^ 

,;<Tbe latter point leads to our next tenet. Evaluation must be used as an 
objective^elenpent of thcdecision-making-^process^ ..jAsL^an aid to decisiofl-making, 
an evaluator (often th^;^ vocationdi director) must determine the areas to be 
examiaed, sources of data, methc^s of analysis and^evaluatioa procedures to be 
employed ln this process. ' '~V~ ^ 

An evaluation plan . When viewed as a helpful tool, the evaluation 
process provides a very important base for decisionmaking related to the 
characteristics, scope 'ot^ the operations, and conduct of the ^program. The 
evaluation process provides., insights into what has happened (the past) and 
hints at what could be. (the future^ / To implement these perspectives, at 
least f ive'^aspects of evaluation should be considered in the planning phase. 

First, one must formulate a clear statement about what speVif ipally is to , 
be evaluated. '^Included in those specifications are all of the* related 
activities. This step requires definition and identification , of all programs, 

— ' ' ' ' 

progt^m participants, and other elements to be observed. 



Second, an evaluation design must -be created that meets your specific 
needs* Consideration must be given to time and fiscal factors and the avail- 
ability of 'personnel who 'will be involved in the ^vaTiiatiion process. 

.Tltird;^ a decision is needed to determine i^ow theVdesign plan is to be 
implemented. ^ . * ^ * 

aFourth^ there is need ^to plan^ow the findings will be reported, to whonk 
they will be reported and what deadlines mu^^ be met in producing and 
disseminating the final report. ^ 

Finally/ a procedure ^i3iiust be instituted to provide feedback to 'thos'e 
involved ii^ the ^prograin operations. *rhis mechanism will help^ to insure 
'^'positive attitudes. tow*ard evaluation* ' ^ . 

The above overview^ of the genej:al phases o£ an evaluation program leads 

to the definition ^ your program. This usually, consists of ; brief descrip- 

*^ • 
tion ^f each vocational program -being evaluated with already established and^ 

^ ' ^\ _ ^ 

defined program objectives. ' * 5 < ^ ^ 

^ . . ^ ^. ■ * f» 

' 'Program objectives are often deriv^Jd from^gen^ral criteria—either stated 

. * ^ )■ 

or assumed. The .latter ultimately determines the type and structure of eval'- 

nation model most su-^'ted to your needs* To aid in tile identification of 

critetia, we will provide- a number of evaluative criteria* You jriay chOose or 

adapt those most suitable to your situation**" In this manner^ you can build 
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vour own models tailored specifically to your program needsT Our plan will 

also provide elemetfts for building of models which may be- applied to depart* 

mental^ module and course evaluations.^ Thus» we are building this monograph 

as one which focuses on user decision*maloing* You must ^decide .on those ele- 

menti* which will help you to expand ^^ur own evaluation potential. ' 

Evaluation As flannjny ^ « ' ^ * " ^ ^^^-^ 

j^rior to illustrating evaluation criteria models or designs', we would 
• ^ * ■ 

like to note that ^evaluation can- also be a planning^ technique (decision- 

0 ^ - - 



makingji^ if you wish).. * * ^ ' " ^ 

Plaiiningi according' to JCenneth H. Hansen (1967)^ has at least sijc 
elements which t^nd to be uniformly agreed up6n: (1) identification of^ ]the 
problenii (2) analysis of problem components i JfS) statements of solutions ^and"^ , 
alternatives , (4) soi»lJtion tests for reality, (5) esuablishment^ of the 
or^nization ' to accomplish the changes/ and (6) implementation of change 
decisions, These six elements, are similar to the so-called scientific met'hod . 
of inquiry. ■ " , ^ , " » 

One major problem^> in vocational education planning is to identify the 
procedures which may be, used in programmed or sequential steps to initiate the 

6 ■ ' 

.previously ideatified' desired changes. To this ead, there are several system- 
atic^^ planning methods that can be^Used. Al^ program plans'^tend to require 
some type of -*needs ^sse^sment." The exact dete^ination on the kinds of 
n^eeds and the .assessment to take plac^e is traditionally accomplished in an 
^.ntuitive manner* HnveVer, if significant feedbacjc is to be providedi> the 
requirement mandates that' planning procedure^* mov^ from an intuitive mode to a 

^ '. ■ . ■ ■ • ' . ■ \ . ' ■/ ■ ^ 

rather highly organized and systematic iriethoif. ^ \ 

We caution tbat!^ "needs .assessmeuits" must essent^ally^^be recogni'zed as th^ ^ 
interpretation of "wants i^* V/heh^ persons ^re polled to obtain a^ list of 
"vocati6ual needs," the usual list will be those activities, skills, , 
c<Jmpetencies,^ or concerns t^hat'are desired by the respective responS^tjs^ ^ ^The ' 
"vocational wants" list must then Be translated to a "nfieds" list; that isj- 
how do the desired ends (wants) become operational,. 

Evaluation (and often .the planning)' of vocational education programs 
traditionally relies on C\tf.xnal evaluators such as accreditation teams ^ ad 
hoc visitations. teams wV^ ara invitid to, examine the totality of a pr6s,en^:ed 
programi or a lor^jl advisory, committee^ /The reports of these teams are, by 



and large, devices which tend to focus on general conditions but not o|i 
feedback/ 
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The reasons for making the above sweeping indictment are as follows: (1) 
the accreditation reports addresiS themselves to generalities; (2)_the criteria 
by which programs are judged tend to be very general ; (3) the intermittent 
cliaracteristic of advisory committee reports means that at least one or more 
years will transpire before there is concerted preparation for the follow-up 
Visitation; (4) in most instances^ the advisory committee members rely on data 
which a^e collected^ presented^ and interpreted by the ^valuatee'-hardly ati 
objective \source; (5) meaningful base line data are not accumulated} nor are 



they maintained on a year by year basis; (6) the teachers^and vocational 
directors seldom, .pay serious attention to a report unless it is so gjtossly 
negative that^ it would threaten accreditation or someone's, job; and (7) 
feedback is seldom directed -to specific components of the vocab;jLonal program. 

To alleviate' the shortcomings of nonsystematic^Iongitudinal evaluations 
whic^ neglect planning} we again stress that vocational evaluation must be 
viewed as a continuous process-'-one that^ in reality ^ proceeds without end^ 
^ We quickly recognize that any evaluation is usually composed of two main 
kinds of data: (1) objective descriptions and (2) value-l^iden judgments.^ The 
importance of the first^ of these cannot be underestimated. The validity of 
the value judgment is dependent in large part on the accuracy of the informa- 
tion" used to make these judgments. In addition^ objective data collection - 
also affords a basis for def:ision'-making and a means to^ plan for implement 
tation of value judgjnentr, 

^ John K. Hemphill (1969) established a series of charactetistics which 
described the process of e^luation, HempbilVs list of six criteria which 
-pertain to evaluation' are listed'^Oelow, 

1. ' The probl^ is determined ^by the situation and t^ecause of its com* 

^ - -. - ... .._ — ^ 

plexity may involve many definers, \ 

2. The task, of evaluation is to test generalizations rather than a set 



of specific hypotheses. The absence of verifiable and^ empirical knowledge 
must often be filled by relying on judgment and experience. 

3. Value judgments are appropriate at all stages of an evaluation study. 

4. Each evaluation study is unique to a situation and can seldom be 
replicated* 

5. Data collection is determined by feasibility and value judgment* 

6. Randomization is extremely difficult or impractical to accomplish* 
Only superficial or selective control over the> multitude of variables is 
possible* 

When applied to vocational education evaluation Hemphill's six criteria 
illustrate a pragmatic view based upon consideration of specific situational 
goals and conditions, i.e*^ local conditions. 

Subjectivity is an element of any evaluation. Th^ "art" of evaluation is 
to blend the objective and subjective for the betterment of the program. WKat 
could be included? Let us examine that question next. 

SELECTED VOCATIONAL PROGRAM EVALUATION CRITERIA 
Several sets of criteria for inclusion in any general program evaluation 
are listed below. Choose as many or few criteria as you deem necessary for 
your evaluation plan* To p'tovide for a complete^ yet not totally comprehen-* 
sive, program evaluation, criteria are listed from several areas. The 
criteria may be. compiled and then converted into an evaluation instrument 
which could serve as an effective tool for measuring various processes ^ 
products ^ inputs and outputs. In this section we have identified only 1 1 
program areas. These eleven general areas include selected criteria for the 
following: * ^^ . 

1. Administrative procedures 
2* Curriculum design . . _ 
3. Staff development 

-6- 



^ 4. 'Articulation and coordination 
5* Liaison activities 
6* Student-related activities 

7- Recruitment and advising ^ 

8* Placement and follow-up 

9. Expanded opportunities ^ p 

10. Facilities and equipment 

11. Evaluation 

\^ile these areas could be expanded greatly^ we are providing a few 
criteria whicb seem general enough to warrant universal application. In later 
chapters of this monogj:aph| we will address specific processes and some* 
procedures by which to evaluate courses i modules i projects and proglrams* 

The criteria which are listed below could also act as a check list with a 
continuum of responses so that you might determine a relatively accurate 
profile on a T)road spectrum of vocational program elements* Or, y6u might 
simply use the* criteria as a "yes** or **no^* list of functions that are or are 
not currently performed* 

Criteria for Administrative Procedures 

^ . 

1* " The procedures by which Jprograms may, evolve are written and made 
available to students » teachers » administrators and counselors. ' ' 

2* Appropriate ^^dyels of resources (moneyi spacei time) are planned and 

; " . ;■ : 

allocated for all vocational- technical programs.^ J 

3* , The decision-makinfe environment exemplifies an interaction among 

administrative staff, counselors, vocational . teachers and other subject 

l:eachers. 

4i Vocational programs are supported by the board, administratioa, 
counseling staff, teachers and students at least equally to other major school 
programs- 

\ ' — -fr — - _. 
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Criteria for Curriculum Design 

1 . Vocational program ^ objectives are systematically re-evaluated to 
determine if changes are needed to reflect the current state-of-the-art^ 

2. Program obj^ectives and re<}uirements are known and supported by the 

V 

faculty and other students, 

3. Program projects have relationships to other subject areas- 

4* Opportunities exist for integration of both females and males into 
the totality of all vocational training programs. 

5. Opportunity is provided for^ompletion of a defined program for those 
entering the .program, 

6. Curricula are kept current through surveys^ advisory committee feed- 
back and community employment opportunities. , 

7. Consideration is given to Jfactojcs whicb.prevent-Students. from-.r nr- . 
pleting programs — absenteeism, tardiness, behavior problems, ^ 

8. Instruction is geared toward individual development based on some 
individual assessment or needs and abilities/. 

9. Program goals include ethical staiulards and practices, 

30* Communications and human relations are integrated within instruc- 
tional subjects . 

11. Vocational youth organization activities are included In the instruc- 
tional and vocational experience programs, 

12, The Work ethos including skills^ knowledge, attitudes, and pride in 
quality work s^urrounding any specific .occupational segment are found in class- 
work, laboratory or experience settings. 

I 

33* Teachers regularly visit training sites to Iceep up-to-date, 

14. Written training plans are developed for each student which account 

for the individual^ general, or vocational objective and proposed occupational 

experience program."^ 

16 .. 
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15. Individual student program outlines ^(objectives) cau be altered or 
changed- .to._J>e ..brought., into ..line with performance through student* teacher 
conferences* * 

16. Actual experiences in the occupational field are provided through 
cooperative. efforts or simulated settings, 

17. A combination of directed vocational experiences and/or simulated 
experiences are provided in appropriate^ clinical settings. 

I8« Each student may choose an appropriate course of study with coopera-* 

» u 

tive guidance and counseling services being offered. 
'Criteria for Staff Development- 

1. Administrators participate in local business, civic and labor organ- 
izations . 

, 1. — Inservice programs attempt to integrate both sexes for continual" 
^^^I'ofessional development* ' 

3- Teaching staff^liieet appropriate state certification requirements for 
*'the special areas in which they teach, have recently completed occupational 

expefience, . are proficient in tho&e skills being taught, and engage in pro* 
fessional growth activiti^es* 

4- Inservice education and staff development programs are provided on a 
regular basis to provide for teacher assistance and training in areas- of need 
or those emerging as^new areas- 

5* First aid and emergency measures are offered as part of the continual 
staff development process. 
Criteria for Articulation and Coordination 

1- Complemehtary objectives, and ^oals are developed within the total 
educational program. ^ ' , 

2. Teachers have support of specific advisory committees for the course 
content and vocational techniques being utilized. 



3* A transfer system is available to ensure smooth transition with 
little or no loss of credit from one school or program to another* 

4* General education aspects of the educational program are articulated 
within the vocational program* 
Criteria for Liaison Activities 

1* Conferences for students, teachers oand parents are a part of the 
program* 

* 

2* The Advisory Committee provides up-to'-date information on current 
trends and develc^ments i^n the community as they relate to vocational -technical 
programs. * ■ ^ 

3; Advisory Committees with adequate representation of\ the vocational" 
technical fields _are organized,. ..functioning, establishing standards an^' 
designing evaluative criteria for the program. 

4* Advisory Committees help to place students and graduates into jobs* 

5* A 'community occupational employment assessment is conducted system*- 
'atically to determine en^loyment opportunities* 

6* Employment trends and needs of the community are Actively reviewed by 

<^ 

vocational staff which include teachers, counselors, administrators. 

V - - ^ ^ ^ 

7* Support for vocational programs comfe^^^*^i?Qin employees and business ■ 
agents in the form of reconmtendations, employment ahd other suppoxt^* 

B*. Resource persons fro|n all vocational areas ar^ invited to assist in 
acquainting students witlj^career opportunities* 

9. Vocational teachers act as liaison persons with the community and 
assist in conducting community relations* ^ 

10. The program attempts to develop personal and occupational develop- 
ment, efficient and safe work 'habits, and positive attitudes toward work, as 
well as sound employer-employee relationships. *" 
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Criteria for Student Related Activities 

ft 

1* Students are encouraged and prepared to participate in future follow* 
up surveys* 

2* Extra-*curricular activities are available for vocational students as 
well; as othgr students- - .... 

3- Opportunities, are provided for student leadership development in 

• ■ - ^ 

J -- t 

conjunction with related vocational organizations. 

ft 

4» Activities of the vocational youth organizations tend to be planned, 

I - 
implemented and e^luated by students* 

■ 

5* Vocational youth organizations are open in membership and particr^ 
pation^ to all students regardless of sex or racial origin. 

6* * All student organizations have supporting and advising services of 
the faculty and administration* \ 
Criteria for Recruiting and Advising 

1* Teachers and students have sufficient understanding of tests and the 
meanings of the results so that the advising process is enhanced* 

2. Appropriate vocational counseling assistance is available for 

it 

students so that aspirations, . interest^,' aptitudeSi and personal physical 
limitations may be reviewed against the availability of occupational choices. 

3. A system of referrals to vocational-technical programs is developed 
^^vith teacherSi counselors and community members* 

4. Career occupation information, including vocational-technical" occupa- 
tions, is developed and made available at all appropriate grade levels. 

5. Any conference involving students, their family and instructors^ is 

* 

summarized and filed with appropriate confidentiality* 

6. Recruiting methods are utilized to insure a continuous flow of 
students with appropriate interests and aptitudes which can insure full util- 
ization of the vocational resources. _ ^^^^^^ 



ERIC 
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Criteria for Placement and Followup 

1» Students have access to the job market" through a wide range of entry 
mechanisms « 

2* Placonent services, available to all graduates and aluraal* are 
-organized-aad' provided by the school* 

3* A relatively high percentage of the graduates are placed in the 
vocational field for which they are prepared* 

4* Job opportunities are listed and updated so that students may utilize 
the information. ' 

5* Follow-up surveys of graduates ^e conducted to determine employment 
status* 

Criteria, for Expanded Opportunities^ 

"*"^l^*.,^^^ontinuing education classes and activities are provided to those 
already in the^work^ force- 

2* Access to ongoing^ocational programs is available for those wanting 
to re-enter the work force or to upgrade theTr^vocational skills* 

3- Provision is made for the full acceptance and integration- of _adult 
students returning to the educational scene. 

4, Open access is provided to all vocational programs for those class-* 
ified as handicapped- Appropriate support and special services are available 
to ensure an adequate success rate for handicapped students. 

5, * Open access is provided for all students from ethnic minority groups- 
v6. Open access is provided to women for training that has been trad- 
itionally provided to a male audience* ' * ^ 

7* Provisions are made to recruit and integrate women into vocational 
programs* 



^Several of these criteria may be appropriate only at post-secondary 
institutions. • ' ^ * v 
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Criteria for Fa ^^i lities 'and Equipment 

1- The ^supply of equipment (including tools, supplies, machines) is 
sufficient to conduct a quality program. \ . ^ 

2. 1 Physical facilities, equipment and instructional materials are appro- 

• ■ j' * ^ 
priate.to the occupational groups included in the program. 

1 ^ 

3l Adequate working space and storage facilities are available to each 

' 'i ' ' 

st-adeut. • ' 

i . ■ ■ 

J. Textbooks and other instructional materials are systematically 
selected, current and-.accessible to-students. " 

p. Safety checks and regulations ^e adhered to at all times". 
Criteria for Ev aluation 

* 1 1. Data are obtained from follov-up surveys to provide evaluative 
ctitferia by which to judge educational objectives and improve the quality of 
offerings- , ^ 

I 2. A plan 'for continuing aad systematic internal evaluation i& estab^- 
lidhed, ■ ^ \< ■ ■ , 

r 3* Representative groups actively participate in occupational needs 
|s^essment, program planning and ^evaluation* * 

4, All' levels of participants (students, paxent^, teachers, '^community 
business leaders) are involved in ongoing program evaluations* 

5* Ongoing evaluations are estjblished for objectives, content, methods, 
outcomes and student performances,* 

. 6, Individual- development is 'promoted through the use of evaluati^on of 
indivi^u^l^prpgr^is^ toward .individual goals as well as a coii5>arison with other 
students/ 

7* StudentNfivaluations are conducted regularly on work attitudes^and 
habits, occupafcional^fe^lopment and relationship with employer. 



8* "Feedback!*, and - findings from evaluation efforts^are^jused in- curri- 
Gulom changes. ^ " . * 

In Closin g , — ^ . 

M . The above list is general in scope and^addresses several general elements 
for program evaluation. . The list is incomplete; yet^ it illustrates the kinds 
of criteria that can be established by ^h^h to desj.gn and conduct general 
program- evaluations. Now, let us discuss some selected evaliiation^related 
techniques -which can be applied to vocational-education* 
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CHAPIER 2, 

USING EVAUJATIOK RELATED TECHNIQUES IK VOCATIONAL EDUCATION 



Just as one major component of vocational education is teaching, another 
is evaluating^ This chapter discusses some specific educational. concepts and 
techniques which have inq^illc^tions for evaluation. As was noted in Chapter 1^ 

■ * r * - * 

the« planning process is a critical ele^enjt iii the systematic development of 
any vocational education program. Now^^ we present a few selected concepts of 
the 'evaluation process. The achievement variables^of evaluation will be the 
focus of , the first discussion. ^ ' 

ACHIEVEHEHT VARIABLES . . 

Achie vement Tests ^ * ■ * ' 

Achievement tests are constructed to assess, a student's terminal behavior 
or the expected behavior or attainment of the student *after completion of an 
dssigniHent^ unit^ module or course. - Achievement tes-ts are developed to assess ^ 
the degree or quality to which the desired behaVior or performance takesL 
place. Such test scores ^show a level of present sl^ill* development. Achieve* 
ment tests are contrasted with aptitude tests which measure a_*relationship 
between present performance or behavior and. future acquisition of knowledge 
and/or sKills in a specific area. In short, aptitude tests are constructed to> 
predict future- success in some specific area* In vocational "education, the 
major stress is usually on tjie^here and, now, not the filture; thus, achievement 
testing needs some elaboration. i-. \^ ^ * 

Achievement scores^provide tWo types of information: Xl) the student's 
level of skill and/or knowledge relative to an established Itfvel of mastery^ ■ 
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and^(2) the relative-ordering (rank) of the sCudent^s score in relation to the 

« * 

rest of the class. Robert Glaser (1963) refers to the first type of informa- 
tion as 9riterion*ref^fen£"ed measures ^nd^ states ,that these measures depend 
upon an absolute standard .of quality. This means that the student either has 
or has nqt acquired .the pre-detemined mastery lev^l of the skill. The deg^ree 
of skill attainment is usually stated in some descriptive term^ The second 
type of information is usually referred to as norm- referenced since it com- 
pares a student with other students in a select group. 

Glaser refers to the student* s level of achievement as^ occurring some- 
where on a^Thypothetical *continuuiit of knowledge or skill proficiency ranging 
from zero (The student cannot perform a specific weld, repair a typewriter, 
balance a. budget.), to mastery (The student can perform specific we^d^ 
repair a typewriter or balance a budget.)* 

Generally/ achievement tests are norm-^referei^ed. However^ there has 
emerged yet another major en^ha sis that of criterion-referenced tests. Let 
us examine them, and their logical extension to the concept ^ mastery. 

Criterion-referenced tests (CRT) . Marvin C* Alki^ (1974) 'notes that 
three definitions of criterion-referenced tests ^xist with each being applied 

" ' IS * - 

in varying circumstances .v The first definition is that of Robert Glaser and 

Anthony J. Nitko (1971) who refer to a criterion-referenced test as one that 

is intentionally designed to produce scores which may then be directly inter*< 

preted regarding previously specified performance levels related to mastery of 

the subject.^ These performance levels are established, through jthe definition 

of the domain group of behaj^idrsc^^r sets of knowledge which the student 
^ *^ * 

<«should be able to perform or know at the end of a unit, module, or course. 

,*The second definition is coined by' Major Harris and Deborah Miller 
Stewart (1971) who "write that a "pure" criterion- referenced test is one which 
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contains items dr^wn from the domain or <5lass or behaviors^jd^efined as necess- 



ary for, mastery of the subject* The CRT conjoins only a sample of behaviors 

or sets of knowledge drawn from the domain of^ reference . The students score 

\ " ' ^ / 

indic^^es the level of student success ^f mastery of the domain behaviors or 

knowledge". . " " ' ' 

V< James Popbam and T<R< Hus^k (1969) report a third acceptable defini- 
tion in whichSa CRT is used to determine ^ student's status regarding a per- 

formance standard or a set criterion* ' They suggest establishing the student^s 

■ ■ 0 ■ * 

performance in relation to the subject's mastery* 

These three definitions may differ in the constraints which are placed on 

the^ criterion-^referenced test, ^ but they share three common characteristics: 

(1) common concern relative to test organization, (2) selection of the indivi^ 

dual test items, and (3) assessment in relation to a pre^determlned set of 

performance behavior^ or. criteria* ^ . ^ 

"^ Mastery * Assumed within criterion*- referenced 'tests is the concept of 

mastery. -^Defining mastery has been and continues to be a problem for all 

vocational educators* Mastery learning has been defined in a variety of ways. 

Benjamin 'S. 'Bloom (1968) ■ sjommarizes the basic ijremise by uniting many 

approaches* He observes that if a normal distribution of student aptitude or 

* ' . ■ 

potential is. assuroed* i. e* , a large number of students have average potential* 

Another ' assumption is that a smaller number have either more or l^ess than 

average potential* The' type and quality of Instructipn PLUS •die time allotted 

for learning to occur is adjusted to m^et the individual Aeeds and character- 

istics of these students* If one combines all these assimiptions, then most of 

4 t 

the students should be expected to attain -a mastery level of the subject. 
Bloom notes two types of mastery* . ' 

The first is that the m'astery of a skill or set of knowledge allows the 
^student to transfer" the learning to a new situation* Given a hypothetical or 



real situation where a weld is needed, the student will select the appropriate 

■ ■ * ' ■ ' 

one for the materials and situation. , - ' ' - 



'pasjsed 



,The second type of mastery refers to the percent; or number of items ^ 
on a test, l^e instructor sets SO percent as, the ^mastery level, and 
the' students attaining that percentage on the test are considered to have 
reached mastery. ' * * ^ 

Related 'to these two types, of mastery iaL_the^roblem o^ whether to base 
mastery upon*the selection of a correct answer or the production of an answer* 



An example of* this . problem is" the selection- ^f an a^ccurately^ written or pic- 
torial description of a proper weld as opposed to the actual production of'an^ 
accurate'ly vritten. or pictorial description of a proper weld* "To be consis- 

tent^ if transfer of learning is to take place, then the student should actu** 

■ ^ ' ' ' 

ally be able to produce a proper weld in a variety of situations. 

Other scales * Major L Harris (1971) establishes five measurement scales 

which he felt were directly interpretable without referring to. the scorers of 

* ^ 
, other students. Four of tne scales are easily applicable to vocational educa- 



tion. The maG^tery level of these scales mayftbej. determined on a subjective 

basi3 according to \ individual instructor decisions^ ' Four of Harris^^ scales 

^ i ' ^ ^ . . ^' ^ " 

are listed- below* \ ^ * , " 

1* Bate scale — The amount of. time a student takes to finish a specific 

h 

task, e*gi, 15 mi^nutes to complete a specif task* 

" r''' ; / ^ 

2. , Sign scale — The student can or cannot perf orm^^ a specific tasfc and 

* * 1 f ^ 

does or doesn^t achieve mastery of the task. For Example, can the student 

produce a specific weld o^r balance a wheel, on a single-^item performance test? 

V , ■ ' * 

3* Accuracy scale--The number of times ,* proportionately^ the student 

successfully coiimletes a specific task. For example^ the studenr accurately 



baL^nces a wheel in three of the foui;« times that the task is attempt 



ed. 



\ 



\* Proportion scale — Ihe portion of a group of J^est items, Selected- from 



a previously defined domain of mastfel^ynJerforoahces^in^whid]^ ^ student ha^ 

shown mastery (knows or can perform)* For example ^ the student can perform 

♦ ^> 
thr^e of four different types of welds* 

■'A 

Assumptions of CRT and mastery * Assumed/under the concept of mastery and 

'CriterionrrefeKUced tests is the further assumption that there does exist a 

V 7 

domaiii of correct responses^ performances^ or behaviors. A domain of i^efer- 

/ 

ence for appropriate behaviors relative to _a specific content area needs to be 

thoroughly defined* This domain should not be confused with 'the cognitive, 

psychomotor, perceptual or affective domains as defined by Bloom (1956) , 

Simpson (1966)^ Moore (1967), and Krathwohl (1964). These will be discussed 

later* Carelal delineation of the domain of Correct behaviors assists in test 

construction and student^ evaluation* When using' "domains/^ specific test 

items are referenced directly to the prescribed correct behaviors which have 
« 

been defined as components of mastery. The components of instruction may be 

' * - "~ > . ^ 

linked to match the student learning objectives with evaluation techniques * 

This actiort would be applying the concept of "doma ^- referencing*" 

Problems of mastery . The concept of mastery learning has at least foiir 

unresolved problems: (1). a working definition of mastety does not exist; (2) 

setting some percentage point as a mastery level without determining criteria^ 

for establishing a mastery' standard is arbitrary; (3) ^'ae use of one score ^s 

the indicator of mastery does not account for the wide range of student* 

abilities and need^; and (4) the whole of any one skill cannot be feasibly 

evaluated. . - 

Another concomitant problem of using a mastery approach and also one of 

IJrT is that there is total coavergen^e of the specified learner objectives, 

the performance of the learner, and the evaluation. For' example,^ in voca- 

tiohal educational classes the emphasis is usually on the completion of some 

'*hands on" activity, skill or product* ('^Experiential" is the proper term*) 



•19- 



S7 



VBiile^ftWe have disciissed at gre^t length the use of various written tests 
as evaluation' fc6crLsT^e"nnust— note-^^caatl.Qja . The term "Certified Mechanic'* 
means th^.t a m8chaHic volunteered to take a rigorous written examination to 
test tfis or her kncvledge about the trade. The tests are now nationally 
administered through the National Institute for Automobile Service Excellence, 
a non-profit corporation based in Washington, D.C. The tests developed 
and administered through the Educational Testing Service (ETS) of PrincetoUi 
New Jersey. In addition to passing the examination^ a mechanic needs at least 
two years of experience as a mechanic. 

What we wish to illustrate ^by the above example is that most mechanicsi 
if not nearly all, wotk at Benjamin S. Bloom's cognitive level 3 — application. 
However^ .the test^ being a paper and pencil type^ cannot test applicatioUi per 
^e. Through a series of implied and inferjped assumptionsi if a mechanic can 
pass a paper and pencil test^ then that mechanic can certainly repair, an 
automobile. Quite obviously there is a discrepancy i,* the manner in which a 
mechanic truly beha^J'es on the job. Hechanics repair automobiles; they do not 
take paper and pencil tests when doing so; 

, The creation of actual performance tests whi^ch are converted to paper and 
pencil tests takes a"^ great- deal of time, ^thov^ht, skill and creativity. By 
the way, multiple choice tests .are easily adapted to meet most vocational 
education situations. Our concern is that the students have experience with 
paper' and j)encil problems . ^milar to the testing conditions — prior to ever 
taking the tests! * 

To' do a better job of testing, tlfere are at lease three major aids — the 
50 called ''Taxonomies. " Let's focus on them as means of improving the evalu- 
ation process". 

♦ 

, . Different types of learning result from different learning experiences. 
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A vocational teacher who plans to teach (help the student to learn) various 
outcomes must plan, execute, and evaluate accordipgly. 
^he-^oneept-o£JI^x<^Roniy_ 



The concept vhic^l undergirds all taxonomies as^ decision-making~tools' is 
simply thi^s: not all teacher or studeat* behaviors are ^he same. Some are 
different from others and, accordingly! elicit different jrespbnses from the 
student and teacher. That is, if the teacher acts differently, the student 
'will respond in different wayS. From this we might infer that the student 
then reaming . 

Teachers may be observed by the different actions they perform* These 
actions might be formulating performance objectives , or they might include 
questions to be^asked pr test items to be administered.- Within these^ clusters 
of teacher actious or behaviors (performaace objectives , questions , test 
items) I not all actions are the same. To.illustratei there is a gteat differ-* 
ence between the questions; "When did Henry Ford invent the MQdel T?" and "Did 
Henry Ford really invent the automobile industry?" One way of examining these 
differences is to apply the cognitive taxonomyi often referred to as "Bloom's 
Taxonomy." The taxonomy is basically a classification system which educators 
can use to observei comparei and evaluate performance objectivesi c[uestionS| 
written materials, and evaluation methodologies' (tests)* But, first let us 
discuss th^ concept of learning/ teaching taxonomies. 

What do, we mean- by a' taxonomy ? A taxonomy is basically a classification 

'J - 

system, a way of grouping selected objects together such as plants, animals, 
performance , objectives or questions. But We consider a taxonomy to be somer 
thing more thaa just a classification system. What differentiates a taxonomy 
from a classification system is that a taxonomy is hierarchical in character- 
istic; that is, a taxonomy is a classification system with a hierarchy of 
classes. Not all the classes are at the same level. The method by which the 
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classes are arranged in a hierarchy depends upon the organizing principle and 
the*, type of taxonomy.. 

In the taxonomy of the animal kingdom^ the phyla are arranged according 
to Evolutionary complexity. Thus the phylum Chorda ta (animals with back^ 



bones^ is higher than Porifera (spoQgesyUKich is^hrgher-t-han-Rrotozo gns . In 



Bloom ^;S Taxonomy y the organizing> principle is that of complexity. Higher 
levels^ in the taxoi^omy are assumed to be more con^lex than the lower levels. 

In addition^ the higher levels in Bloom's Taxonomy build upon the lower levels. 

\ . ' ^ 

If a student can perform at the third level (application) then we also assume 
performance at the two levels (con^rehensloh and knowledge) . More on this 
later*^ 1 

Vffaati is meant by cognitive domain ? To simplify matters , educators have 
divided the types of learning which take place in the schools into three 
areas-: ^ Psychomotor, Affective^ and- Cognil:lve . 

The Psychomotor Domain, as you may J^ecall, deals with the manipulative or 
motor-skill \area (printing, writing, wiring). . - " 

The Affective Domain deals with attitudes, interests and values. 
The Cogri^tive Domain concerns knowledge and ^he development of intellec- 
Lties plus some skills . Most of the time, ^ teachers at secondary 



tual abil 
levels an 



concerned with the "Cognitive Domain*' because, traditionally, that 



is how thejr have been trained to evaluate.. 

■ r ■ ' " . ■ 

How can the taxonomy be used ? Teaching can be envisioned as a t^iad of 



acts: Objectives lead to evaluation which is related to the teaching act 
which is djirectljy related to the objective. This process is cyclical and, in 
theory, never endihg. 

-' ■ " i - . ' • ' ■ 

In the abov^ model, the objectives which are formulated should determine 

the teaching procedures and the evaluation procedures. One can use the taxon- 

1 ■ . ...... 

omy in eadh of these processes : in formulating objectives, in developing 
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classroom questions and learning exercises > and in constructing evaluation 
instruiDents or methodologies* ^ . 

Using a Taxonmy in Vocational Education 

^ ' For. educational tools ^ such as the ~ taxonomies^ to be worthwhile^ they 
must be useful in the following ways. You may even wart to add your sug*^ 
^stibus "to-this-list — 

1, Range of objectives * A taxonomy provides a list of possible ranges 
of objectives available in any subject. A close examination of the categories 
may keep a teacher from over-en^hasizing one" levels such as the knowledge 
level. ' ' * ■ 

2, Sequeacing objectives . An analysis of learning tasks will indicate 
to the teacher the learning experiences^ necessary for the student to obtain 
the intended outcomes. The taxon<xiiy provides a means to sequence learning 
from simple to con^lex outcomes* Sequencing also aids in determining the 
order of presentation, 

3, Reinforcement of learning . Since each lower category of the taxonomy 
is subsumed by the next Higher category^' reinforcemeat of previous learning 
occurs if learning experiences are properly sequenced, * . 

' 4, Cognitive structure provided * Facts are presented Co students in a 
cognitive structure by being related to concepts^ applications or problems , 
Students are relating facts to larger constructs instead \>i memorizing 
isolated facts* 

5. ^ Congruence, Once an objective (or question) is written and classi- 

V 

fied at a particular levels it aids the teacher in selecting more appropriate 
teaching strategies and evaluating techniques which coincide with the level of 
the objective* If an objective is written at the application level, learning 
experiences for students _must be provided at the application level* Further^ 
students should te tested ot evaluated at that level* If the goal of a 



particular Vocational Education, course is to teach a person how to asssemble 
circuits, then the teaching activities should be aimed toward that goal and 
the evaluation. should match. Ii\ such an example, a paper and pencil test is 
incongruent with the learned behaviors- 

6. Diagnoses of leamitfj; problems . Should a student not achieve the 
intended outcome at the level specified by the teacher/ the teacher can sys- 
_tematicallv examine at which level the student" is encountering, the learning 



difficulty, and tjiereby prescribe a^iTltin^l^exper^ieace i to hel pjthe student 
overcome a specific learning deficit. 

7. Learning to learn . Students arte able to perceive that learning can 
be sequenced according to the relationship of the categories to each other^ 
obtaining a^^nodel^of learning, j^hicb^ they, too, can use when they leave the 
classroom. 

8. D esigning appropriate testdtems . Teachers who understand the prin- 
ciple of fairness will be quick to use the taxonomy as a" self^evaluation of 
test items* There is evidence to show that most teachers at most levels t^ach 
at rather lovif levels of thinking. Yet, tests are often -constructed at higher 
levels of" thinking. This is not fair to the students* A teacher can match 
l^mer objective with the test item to determine if the test appifoximates the 
level of the objective. This is'an application of the concept of congruence. 

9. Decision'making * By using a systematic method of analysis, *>you can 

decide where the learning will lead and how much time to devote to establish^ 

ing meaningful prerequisite skills. A taxonomy can thus approximate an 

instructional road map* ^ ^ . * 

* * * 

Figure 2^1 illustrates one way '^f using the totality of the domains to 
determine the kinds of functions which are needed to implement specific evalu- 
ative tasks. 
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levels of the Cognitive Taxonoioy 

According; to ^loom, et al., there :are six_ basic levels of cognitive 
learning. They are arranged in descending order from the lowest level to the 
highest: . . 

1. Knowledge*'-reinembering, recall or recognition 
^ 2* Comprehension*— understanding a concept or principle 

3, Application^rusing infopnation in unique situations 

4* Analysis**subdividing an aspect into its component parts 

5* Synthesis — creating a whole from parts 

6* £valuation-*'judgiiig via a set of established criteria 
An Analysis, of the Cognitive Taxonomy 

* Bloora*sTr^onomy.Jias:^^era^sed a am analytic "tool" since 1956 > and an 
evaluation of the taxonomy in relation to clas^rwiin^Tise'-and^Qther related 
research seems appropriate. Such an evaluation would not only point out areas - 
of concern^ but limitations- as well* 

■On the plus side» the taxonomy has gained widespread acceptance in voca- 
tional education and has proven to be a useful tool for curriculum devel- 
opment^ .te:ctbooH writings anci instructional and evaluative planning. 

Though research for the most part has validated the ^hierarchical stxuc- 
tuire of the ^taxonomy, this same research has raised several questions* The 
questions seem to ' be focused on the lower ^and upper ends of the taxonomy* 
However, the questions posed should not overshadow the practical utility of 
the taxonomy* Further research on the taxonotiiy should prove .instrumental in 
refining a valuable educational tool with direct implications for- curriculum 
decision*'making by the classroom teacher* 

This and other cognitive taxonomies provide a convenient reference system 
for vocational teachers at all levels. It is more difficult to provide educa- 
tional experiences at the analysis, synthesis and evaluation levels,* Perhaps, 
■■ * 



vocational ^educators must simply place more emphasis on application than on 
the three higher , levels . "This does not imply that vocational teachers should 
not attempt highei: level areas. Yety we suggest that vocational educators 
attempt the latter three to the utmost since the students should have the 
basic competencies to accomplish higher level skills . 

■ The one major contribution to teaching is. that use of the taxonomy brings 

■ ^ fc ' - ■ ■ - _ 

to the cognitive ^awareness level the fact that teaching and learning can have 
a structure. But> the teacher decides what the structure wtll be* 
The Affective Domain 

Whiie the Cognitive Domain c<Jncerns those "intellectually'* related goals 
of the schools, there is yefef'another domain which relates to feelings^ atti' 
tudes and emotions-*the Affective Domain. The^ latter may probably be more 
important than the former but not ■ in our /Industrially' oriented society* - 
Production and attainment ^re held in higher esteem than is "feeling good" 
about some thiiig->-^ And, following those societal goals ^ we shall merely intro-* 
duce the Affective Domain, knowing fuIIy^eM^that we have done so* 

The developers of the Alfective Domain^ wanted to establish reference 
points in this vital area so that instructional objectives could be developed 
within a systematic frameworlc* Five major areas are described the domain: 

1. Receiving 

2 . Responding . ■ ^ 

3. Valuing . 
4 * Organization > ^ ' 

5. Characterization by a Value or Value Complex 
These five areas are then, subdivided > just as is the! Cognitive Domain. Below 



^avid R* Krathwohl, Benjamin S. Bloom, and Bertram B* Masia* Taxonomy 
Educacional Objectives: The Classification of Educational Goals. Handbook 
II: Affective Domain* New York: David McKay, 1964* 
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is a very brief outline of the entire Affective Domain. You are referred to 
the source for further study and details just as we used it for the outline 
presented here/ * ■ ' . " 

Receiving . The initial classification in the Affective Domain relates to 
a learner* s ability to be sensitive to happenings in the- environment/ In 
short* we want;to provide instruction so that students and citizens can recog- 
nize that some phenomenon is taking place. To become sensitive^ one must 
demonstrate _*'Awareness/' which is the first subdivisj^on of Receiving. Being 
aware is only the entry step^ This is followed hy the category of **Willing- 
ness to Receive", The ^ final subdivision 6f Receiving - is- "Controlled or 
Selected Attention". You "demonstrated selected attention wh^n you decided to 

focus on this -page, rather than the local newspaper, which may be on the same- 

* ^ - - * - * ^ 

desk or table on which this monograph is"pla"ced.- 

* ' ■ ^- * 

These behaviors are all controlled by the individual, "The individual is 

■ " / * " ■ 

cognjLtively aware that tiie behaviors are being emitted." Further, "think for a 
EQoment that^ the" aboye described affective behaviors can all be_ taught, and 
learned. That becomes one major contribution of the Affective Domain., 

Responding . The second major element in the Affective Domain is called 
**Respondinp*T~^'i^hile,.^^one^may be willing to' receive, and the like, that 
behavior needs an action con^onentT^T^ behaviors are demon- 

strated by. actually engaging \in activities which- reTate'^p -.Receiving. 
** Acquiescence in Responding" i^ the^^i;:st subdivision of Responding which is 
demonstrated when one complies to re^gulations or conventions, ^^Willingness 
to Respond** is the second ^subdivision. The third subdivision in the^major 
category of Responding ip "Satisfaction in Res^nse/'" 

Valuing . The third major category in thfe Affective -Domain is called 
"Valuing". As may be connoted from the temii it means that one internalizes 
the concept of "worth.'* lijhat sets this category in pro^r perspective is" that 



the Valuing is. exhibited by the individual as a motivated and deliberate 
action or frehavior-mot simply thb villingness' to acquiesce. 

There are three subdivisions to Valuing: (1) ''^Acceptance of a Value^" 
(2) ''Preference for a Value" and (3) ."Commitment'*. These three" subdivisions 
are nearly defined in their titles. , g ' 

Organization . As a learner's experiences broaden, there comes a point 
where values.begin to be ordered, classified or arranged. When such behaviors 
occur, then Tthat- individual is . operating at the fourth major category of the 
Affective Domain — "Organization". Within Organization are two subcategories: 
"Conceptualization of a Value" and "Organization of a Value System". 

Characterization by a Value or Value Complex . The^ hi^est category of 
the Affective Domain is a demonstration of behaviors that shows that an in- 
dividual acts, on a rather consistent basis with those internalized values 
which the individual holds In Affective Domain talk (almost like CB jargon) 
acting as one believes is described as Characterization by a Value or^ 
"Value Complex". The first of two subdivisions within the broad category is 
"Generalized Set," consistent, actions or commitment to attitudes, beliefs oi: 
values^ The .final subdivision is "Characterization,'^ which implies that an 
individual is completely subsumed by the v^lue. ^ ^ 

, We realize that our treatment is highly abstract. But If you will only 
.ponder on the implications for the schools and the impact of your field on "the 
total environment of the schools j ybu will quickly generate list? of activ- 
ities^ .whicli could accompany those cognitive or intellectual objectives. As 
RoberT^FT^Mager wrote .in the front piece of his book Developing Attitude 
Toward learnin g (1968), '^'Tf^I .dp little else, I want to sen<i my' students away 
with at least as much* interest in^^thT^^sul^Jscts^ I teach as' they had when they 
arrived.". " 



A Vefry Brief Analysis \ ■ * 

Recognizing that we'' bavKnot adequately treated the topic, we will, at 

any rate, attempt to^analyze the Affective Domain as" it relates to i^istruction. 

In our opinion, this domain tend^ to generate far too many subcategories for 

the typical teacher to use. , We also infer that the di(^fferences between sub- 

divisians are too artificial . Some of the subcategories ^eem to be Very 

similar to^selected cate>gories of the 'Cognitive Domain. Finally, the time 

that IS needed in the schools ' to provide all the ne^ce^sary experience? for 

both the cognitive and affective dimensions of learning would be overwhelming. 

^ Yet, we, can counter, "VBiat are schools fot?" The kinds of work and 

vocational attitudes that are developed in vocational education certainly 

affect the students* Our attitudes toward learning are school related* Our 

attitudes about interacting ^with each other are ^shap^, in part, in the 

* 

'schools* The, belief in one's self is highly school related. All of' these 
aspects are more* important than leaniii?| and promptly forgetting the differ- 

' ence between transitive and intransitive verbs.' 

Perhaps, the very be;st of our vocational teachers subtly interweave both 
the cognitive and affective consequences in ' their instruction* Recall that 
there is a high g^rob^bility that the teachers ^hom you lik^d best, just' also 
happened to teach the subjects that you also liked best. >^ / ' 

\ie ^don't.have the answer to the perplexing problem of merging the Cogni* 

* ■ *, * ^ 

tive and the 'Affective . components of _ instruction : Perhaps one of you now^ 

r^eading this paragraph may synthesize and publish that solu£lon! ^ ^ 

THE PSYCHOMOTOR SOMAIW 

One of the newer domains *co join the taxQnomies is that of the Psycho- 

motor Domain which articulates movement instruction in a .^y^tematic manner. 

.There are several classification systems developed in rthis area, while there 

are few in the previous two ^doj^yains * The system which we chose to illustrate 
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is that which the American Associatioik for Healthy Physical Zducatioiii and 
Recreation (AAFHPER) has produced: 

Ann E/Jewett and Marie R. Mullan, '"Movement Process Categories^,** 
.frpm !*Movement Processes in Physical Education Teaching-Learning.*' 

In, Curriculum Design: Purposes and Processes ilf Physical Educatjjon ^ 
. Teaching^Leaming ^ WashingtoUi D. C: American Alliance for Health, 

Physical Education and Recreation, pp. 9-10, 4977*" 



■ One miataken "n'otion that the more ^^intellectually*' oriented teachers'^ 

often have is that psychomotor skills are of little concern to learning cogni- 

.tive tasks. We say '*mistaken^* because, if one analyzes the ways in which wq 

learn, .they are anything but static. We move about, coordinatii^ " hand*eye 

^movements^or lejg^hand-eye'-brain actions almost continually. Xhus, there ought 
'* ■ ^ _ ■ , ' 

to be included within the curriculum means b^ which to organize the'physical 

movements associated with living in a systematif^c manner. « ^ 

The* conceptual framework of Jewett and Mullan's Psychomotor Domain pro- 
gresses from individual, needs* to those of social interaction: 
* 1. Fully developed humaris. - ' ^ ^ 

^ 2. Ability to control and adopt tht physical environments 
3, Ability to relate to others in interactions "^and culture 

The Classification Scheme ' ' • 

'. ^ . " - 

Below is quoted the general areas of the ^'Movement Process Categories** of 

the Jewett and Mullan psychomotor d<Hnain with the permission of the American 

Alliance for Health, Physical Education and Recreation; and the a^tthors. Note ' ( 

the ease of application of these processes to vocational education skills. 

A*^ Generic Movement: Those movejaent operations or processes which facilitate 
the. development of characteristic and effect^ive motor patternsi ^They are 
typically exploratory operations in which the learner receives or "takes in** 
data a^ he- or she moves. t ^ * 

41 

1. Perceiving; Awareness of total body relationships and 6f self in 
' motion* .These awarenesses ma^ be evidence;^ by body positions or 
motoric acts.; they maV be sensory in ,thaV the mover feels the equil- 
ibrium of body weight and the movement of limbs; or they may be 
evidenced cognitively through identification, recognition, or differ- 
entiation.. ^ ' ' • * 



/ 
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2. Patterning: Arrangeroeikt and us,e^f bpdy parts in successive and 

harmonious ways to achieve a movement pattern cr skill. This process 
is dependent on recall and performance or a mo^^ent previously 
demonstrated or experienced. 

* ■ 

B* Ordinative Movement: Thfe processes of ^organizing, refining, and perform- 
* ing skillful movement. The processes involved are directed toward the 
orgahizaition of perceptual*motor abilities with a view to solving parti;^ 
cular movement tasks or requirements; ^ ^ 

^l* Adapting: Modification of a pa^tterned movement to meet externally ^ 
imposed task devahds. ■ This would include 'edification of a part:icular 
movement to perfom it ornder different conditions. *■ 

2. Refining: Acquisition of smooth, efficient control in' performing a, 
movement pattern or skill by mastery of spatial and teijpporal rela- 
tions*, This process^ deals with the achievement of precision in ^motor 
perfprmanc<± and habituation of performance undeir^ more com^ileK^ .conditions 

Creative Movement: Those motor performances which, include the processes 
cf inventing or creating movement which Will ^rve the persona^ (individ- • 
ual) purposes of the learner* The processes employed are directed toward 
.discovery ,'^integration, abstraction, idealization, emotiohal qbjectifica- 
tion and composition* " i * ^ * 

1*' Varying;, Invention or construction or^ personally unique optioss itf 
motor "performance. These options are limited to different ways of^ 
performing *^pecific movement; they are of an immediate situatibnal 
nature and'lack any predetermined movement behavior which has been 
externally imposed on the' mover. 

:\ 2* Improvising;^ Extemporaneous origination or initiation of personally 
* aovel movement or combination of movement. tThe pi*ocesses involved 

may be stimulated t^y a situation externally structured, although con- 
. scious plan-tijfig on the part of the performer is notmsually required.' * 

3* Composing: CombiU&tion of learned movement in personally unique motor 
designs or the invention of movem*ent pattfemij new to the performer. 
^ The performer creates a motoif response in terms of a personafl inter* 
pretation of the movement situation. , ' ^ 

Conclusion , ' . ^ ' ' 

The use jof the taxonomies as aids to vocational evaluation are critical. 

If one teaches at one level and tests or tvaluates at another*,, t^en the stu- * 

dent is beifhg abused* *^^urther, the type of eval^atiod beings u£;ed should 

reflect Ti* scope and sequence of skills whi-ch are. hierarchiTcally ^arranged. 

Thus, vocational educators do have a theoretical framework oTi which "to b^ng*' 

their objectives. Our next 'chajjter will address the . concept of evaluation 



models so ^ that the process > may be viewed from even a greater perspective. 
Finally^ we n it caution that while the domains tend to be ^theoretically 
oriented, their applications can be universal in vocational education. For 
example^ the current **brief enthusiasm" for accountability is almost totally 
inafipropriate for vocational educators. While there is a very strong intel* 
lectual component in vocational education^ that contponent is not the only 
coinponent. Thus^ the cognitive domain should receive* equal attention with, the 
affective^ and. cognitive domains by vocatioual educators. 

Employers desire persr^s who have good attitudes^ who can work indepea- 
dently^ who .can get' along with others, who show initiative and who can 
corapleteia job withpride. 

Vocational educators have long kno^m about the above traits. In most 
instances, the "affective" skills are taught to persons. enrolled in' vocational 

ft ^ i 

courses or programs. So we caution you — do not abandon the tr^itional stress 
on values ,and skills for the "ease and convenience*' " of ' cognitive test 
profiteers , ^ ^ . ' 

Now let us examine one set pf those evaluation models^ / 
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Chapter 3 

^ . ^ SELECTING AN APPROPRIATE EVAUUATION MODEL 

The goal of *all education! given adequate available resources, is' to 
offer the iiighest quality program possible* If vocational administrators and 
teachers are to' maximize the quality of their programs, they^ must ""lect 
tredibl^ information so that their decisions will also reflect well — in a' 
qualitative .sense* We now focus dur attention on the selection of evaluation 
models by vbich vocational educators may assess j;heir 'programs. The task^ 
which confronts the vocational education* evaluator is to f elect from among the 
many »7ailable alternative models* 

A REVIEW OF SIX MODELS 

We have selected six vell-knovn modj^ls for discussion* Examine each to 
observe their commonalities* With such analysis, you can produce yoMt own 
eclectic model* The ^ix models vhich'^will be presented are: 

1* Accreditation Model 

2* ^Tylerian Model 

3* CIPP^ Model ^ 

4. CSE Model * . * ^ 

5- Formative and Summative Model 

6. Single Subject Design Model 

I 

As you may recall, in Chapter 3 we stated that any conducted evaluatitm 
must ultimately lead to better decision-making. That common thread will be 
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woven throughout the .six models. Evaluation should not be perceived as being 
a mechanism by which to fix blame or to be destructive. If any evaluation is 
to be helpful^ it must be viewed as a formal process by which programs are 
improved through improved decision'*making which is based upon objective data* 
Now^ on to tho'se models. 
The. Accreditation Model^ 

if 

Accrt^ditation represents tha oldest type of evaluation activity. ' This 
model typically uses the role of the '^expert observer." Emphasis is placed on 
the processes or means of education as opposed to consequences of educational 
activities on the learner.. Procedures have beea developed by such organiza- 
tions a$ the American Vocational Association^ National Council for the Accred* 
it^ation of Teacher Education^ and the North Central Association of Colleges 
and^ Secondary Schools. Accreditation reports are used to identify deficien-* 
cie& in the education of students and their teachers and to withhold accredit 
tation of programs or place embargoes on graduates if deficiencies are dis-* 
covered ^,s^nd are not corrected. The variables examined in evaluating the 
programs typically include such criteria as those emphasized by the North 
.Central Association: . ^ ■ - 

1> The '^general intellectual and moral tone*' of the .school 

2. The school plant ^ * 

3> Instructional equipment and supplies ^ 
^ 4. The library and its services < 

5. Financial data and personnel records 

6. Policies of the school board 

~~ ■ ' 

^The authors wish to thank Dr* William P> McDougall and Dr> James T* 
Shoemaker of the Department of Education of Washington State University for 
their generous assistance 'and permission to use some of their previously 
written materials afbout the Accreditation and Tylerian Models > 

_-36- 44 



7* Organization and administration of the school 

8* * Teacher qualifications (degrees, subject matter preparation) 

9* Teaching load 

10* Whether the curriculum meets pupils^ needs and interests 
11* Guidance services 

12 * The school as educational and recreational center for the entire 
community 

Implicit in this approach is the assumption that the quality of the 
facilities » the numher of course units in the curriculum^ and the number and 
kinds of courses taken by the teachers have a significant effect on the qual- 
ity of vocational education* . 

High degrees of standardization have evolved in the development of our 

* <•' 

ft 

accreditation procedures and the utilization of systems describing such varia- 
bles as training requirements » course units » plant facilities and finance* 
Little attention is paid to direct evidence of learner behavior. Gene V* 
Glass (1969) has observed that instructional outcomes are wedded to the 
faculty theory of psychology in vogue in the early 1900* s* He noted that, such 
theory is quite prevalent in pur curricula* We would add that the "acpount- 
ability*' movement is equally as misguided* Such psychology persists in large 
part to this day as far as the traditional approach to accreditation is con- 

cerned^ an assumption which is not now universally accepted but one which uas 

* ' 

gone unexamined in many accreditation practices* The Accreditation Model has 
borrowed little from. the behavioral and social sciences* 

Overtones Qf this model chn quickly be recognized as having direct par- 
allels in our traditional jsatterns of teacher certification* An accreditation 
visit embraces almost precisely the variables whidi are reflected in most 

standard certification .patterns.^ These include the number of course unibs 

" t ^ 

teachers have fulfilled, extent and reputation of the training institution, 
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degrees of faculty, and size of .library, Little attention has been given to 
the direct assessment of "learner behavior except as it is incorporated in 
standard course *p^atter^s . 

A major weakness of the Accreditation Model is that standardization and 
wide acceptance of a process does not guarantee its effectiveness. Processes 
Daust be continually validated through experience and research as reflected in 
learner behavior. Counting and tabulating institutional artifacts may indeed 
be an exercise in. futility if a direct link to learner outcomes is not present. 
Standards against which vocational education programs are evaluated cannot be 
totally arrived at through deliberation of experts on the subject but rather 
must depend, in part, on direct assessment of the performance of the learner. 
Other deficiencies were noted in Chapter 1, ^ 

The Accreditation Model is incorporated in most state education agency 
evaluations. Thus, we must address the model as one which will probably b^e 
around *£or some time. 
The Tylerian Model ' 

The Tylerian Model was orj.ginally devised as. a curriculum evaluation 
model during the 1930* s for the classic "Eight Year Study.*' The basic method- 
ology was presented by Ralpff W, Tyler and his associates (Smith and Tyler, 
1942j Tyler^ 1951) and has had broad application and influence in all areas of 
education, 

Tyler's. Model places almost exclusive priority on learner behaviors; It 
is the ends of instruction and not the means, or processes which are important, 

r 

V 

The stej^s in the Tylerian Evaluation Model are to: 

K Forhtulate objectives , " Determine the broad goals of the program, 

2, Clas'sify ob^j^ctives . Develop a typology of objectives so an economy 
of thought and action may be achieved, 

. . ■ * . \ 

3t Define objectives in behavioral terms . This feature has become the 

cornerstone of the Tyler Model, '^Modern" methodologies of evaluation 
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which rest heavily, upon the specific, behavioral statement of objec- 
tives have not moYed beyond Tyler's thoughts on evaluation in the 
Eight' Year Study. 

4. Suggest situations in which achievement of objectives will be 
shown. 

5 . Develop -or select appraisal techniques (standardized tests» ad hoc 
tests.» questionnaires > 

6. Gather and interpret performance data > The final step in the evalua" 
tion process involves the measurement of student perforEtiance and the 1 
comparison of performance data with behaviorally stated objectives.' 

' . ' ' ■ 1 ^. ^ ; • 

The Tyler Model has had pervasive influence on eiJucational;. thought *the 
past few years. It has served to help shift the focus of- our tb^inking frora 
process ta product an<i to make explicit the need for clearly defined goals in 
terms^ of learner behaviors. Though dirept learner influence' is difficult to 
assess » it is probably the Tylerian Model which has encouraged educators to 
incorporate more current research from the social and behavioral sciences into 
the methodology of educatioa. Tyler's efforts have lead to the model being 
utilized, in packaged learning approaches^ accountability. by pbjective plans^ 
as well as many federal program evaluation efforts. The key emphasis, as it 
relates to the "processes ^of vocational evaluation is the direct definition ? d 
assessment of learner behaviors. - . 

Although the Tylerian point of view was initially creative, it \\as almost 
evolved into a raass (mess) of quickly written behavioral objectives. After 
years 10 f experience with'^this model, there are unsolved problems which con- 
tinue to, reoccur. Four of those unsolved problems follow. 

For a significantly large portion of vocational Education curricula^ the 
actual behaviors that are desired cannot be observed directly* Training and 
instructing must be evaluated through classroom tests, by observation in. role 
playing situations or simulated exercises, and through the learner's Written 
or spoken reactions. The?i& behaviors do not fully represent real performance. 

Behavior mpdels and instrument development and validation are still at 
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primitive stages. Definition and measurement of most psychomotor behaviors 
constxtute-major stumbling blocks. /' 

Research to establish desired behaviofs for various vocational purposes 

jt * 

is incomplete and^ indeed^ must be a continuous and evplving process. 

The^Xyier Model has been used now for nearly one-half a century. Despite 



the fatt^that it-^&till has ardent defenders, it alone is inadequate as a mod 
for '^aluatjuMAC^utcomes of in^tmction and training. -It is ill suited.^^£ot 
evaluating : problems of vocational organizational planning, facilitates ,V^nd 
equipment, program rationale > or financing. It has and does, make a signi*ii* 
cant contribution to educational thought in that it emphasizes the .central 
importance of learner behavior/performance. 

CIP^^ Model " Z 

. The CIPP Evaluation Model was developed by Daniel L. Stufflebeam (1970, 

1971) to provide a mechanism to improve both the intended types of decisions 

that were being made about programs as well as the actual decisions that were, 

-in fact, made ab6ut programs. Stufflebeam perceives evaluation as having at 

least two major ends: (1) to determine ^e objectives, i.e., planning de- 

cxsions and C2>'-to judge and react to what bas actually taken place, i.e,, 

recycled .decisions. Further^ the. CIPP Model views as "means" the procedures 

by which decisions^^axe implemented. * 
■ ti . * 

t- 

Thus, the CIPP Hodel is a means by which alternative deci'sion-making to 

* • . ' ' 

current practices is a basic component. To accomplish any evaluation, 
Stufflebeam suggests that -evaluations be continual and systematic processes. 
To conduct the systematic evaluative processes, there are four major evalua- 
tion aspects: (1) context, (2) input, (3) process, and (4) product. You will 
note that these four concepts lead to the acronym of CIPP. Each of the compo- 
nents is used means leading to specific decisio [unmaking actions* 
-Stufflebeam views the decii^ion-making process as that which would modify. 
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adju&t, sustaiiii or discontinue an educational program or.any o£ its parts. 

Context evaluation^ according *^ to Stufflebeaitii is conducted where the 
activity takes place so that, one may gather information concerning needs^ 
problems and objectives. The context evaluation is a ^'reality" check. 

Input evaluation is accomplished, so that data may be gathered about 
strengths and weaknesses of alternat^ive .strategies which could also accomplish - 
the program's objectives. Thus, context evaluation takesjplace to identify 
the best possible objective to accomplish a desired end. Input evaluation 
would then require judgments abou,t relative strengths and weaknesses of the 
procedures whibh are used to implement the objectives. ' 

Process evaluation is accomplished to obtain information about the exact 
strategies which ate Used to implement the procedures. By using^ process 
evaluation methods^ one. would determine the various techniques, strategies^ 
and designs by which procedures are implementec in a pirogram. 

Product evaluation refers to an over-air decision-making process in which 

one would^ continue a project as is, modify the project and continue^'to use it| 

> 

or terminate the project. . . ' 

To' use, the CIPP Modeli^ there are several issues which must be addressed^ 
There is need for personnel who would determine the various specific aspects 
of each of the'se CIPP components. These persons would have to specify exactly 
what would be contained in the context, input, process, and product dimen- 
sions. Second, personnel would have to be identified to collect the data for 

each of the four areas* Finally.i a series of standards would have to be 

t- 

developed by which the evaluation itself would be judged either meaningful or 
useless * 

Figure 3-1 illustrates how. the CIPP Model may be used as a decision- 
making model which stresses a project's objectives and methods in relation^ to 
decision-making. 
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OBJECTIVE 



METHOD 



RELATIOK TO 
BECISIOK- 
MAKING IK THE 
CHANGE PROCESS 



CONTEXT EVALUATION 



To define the operating 
context, to identify, and 
'assess needs ^'anS opiiortuni * 
ties ' in the context, and to 
diagnose problems underlying 
the needs and opportunitieis . 



By describing the context; 
by compajcing actual and 
intended inputs and_outpu_ts ; 
by comparing probable and, 
possible system performance; 
and by analyzing possible 
causes of discrepancies 
between actualities'^ and 
intentions . 



Far deciding upon the set- 
ting to be, served I the goals 
associated with meeting 
needs or u^ing opportu{iitieS|<> 
and the objectives associated 
with solving problems, i«e«i 
for planning needed^ changes* 



I (PUT EVALUATION 



To identify and assess 
system capabilities ^ 
a va i 1 able input sgrat - . 
e gieS i and designs for 
implementing the 
.strategies. 



By describing and analy** 
zing available human and 
material resources » 
solution strategies I and 
procedural designs fox 
relevance I feasibility 
and economy in the course 
of action to be taken. 



For selecting sources of 
support ^ solution strate- 
gies, and procedural 
desi:gnS| i*e* , for 
structuring change * 
activities* 



FIGURE 3-1; THE CIPP MODEL'S FOUR TYPES OF EVALUTION 



Source: Daniel L* Stufflebeam, "Programmatic Change*" A paper presented at 
the Annual Convention of the American Vocational Association, Ne\^ Orleans, LA, 
December 5, 1970, Reproijuced with the written permission of Daniel L. Stuff lebe'am. 



PROCESS EVALUATION 

« 



moma EVftuJAiiow 



OBJECTIVE ' 



METHOD 



\ 



RELATION TO 
•DECISION- t\ 
MAKING IN THE. ' 
CHANGE PROCES^ 



\ 



To 'identify or predict irr" 
^process, defects in tt(^ 
proceduraj, design or its 
implementation or to provide^ 
information fQr the pre- 
programmed decisions, and 
to idaititain a record of . 
procedural events and 
activities . 



By^monitoring the activi- 
ty's potential procedural ' 
barriers and remaining alert 
to unanticipated ones, by 
obtaining specified infor- 
mation fo;: programmed 
decisions, and by describing^ 
the actual process. 



To relate outcome - infor * 
mation to objectives and 
to context, input, and 
process information. .^^^ 



For implementing and 
refining the program 
design and procedure , 
i.e,,.f6r effecting 
process control* * 



By defining operatio^ially. 
,and measuring, criteria 
associated with the objec- 
tives, by comparing these 
measurements with prede- 
termined standards or com" 
parative bases, ^^nd by 
interpreting the outcomes 
in- terms "of r-cc^orded con* 
text, input, Ifnd process 
information. 



For^ deciding to continue, 
terminate , modify , or- re- 
focus a chrfnge^ctivity, 
and for linking the a^ct|* 
vity to other ipajor phases 
of the change process, 
i*.e*, for recycling change 
activities* 



FIGURE 3-1, Continued:, THE ClPP MODEL^S FOUR TYPES OF EVALUATION 



Source; Daniel 1. S^ufflebeam, "Prograinmatic ChangeJ' A'paper presented at 
the Annual Convention of the American Vocational Association, New Orleans, LA, 
December 5, 1970. Reproduced with the written permission of Daniel L. Stufflebeam, 
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Stu^flebeam views evaluation mUch as we do — that is proactive. Pro- 
* ' *\ ' ' - . * . - 

active ineans^ that the evaluation as conducted so that it may provide info^ma- 

, ' - ^ ' - i ^ 

tioa concerning decision-^making all along a^project's 'route* -The CIEP Model 

; " * . * ^ 

* ' r 

is not a model which can be used simply as a final evaluation technique* 
Finally, the CIPP Model is, oriented to the users. * ^ 

Further, the model could help school programs to be more, credible to* 
outside agencies which, fund vocational education. For^exaraple, if one wgre t^ 
conduct a project sponsored under a grant, from the Restarch Coo rd^itia ting Unit 
of th6 Washington State Commission for VocaUonjSffiducation thefc the CIPP 
Model could be used to show what -took place during the .entii;^ duration'^of the 
project* To be sufce, there would be several technical cortiponents of th\ CIPP* 
Model* 'These would include the identification of what exactly was going to be 
evdluat^ed, the kind of evaluation instruments that you would use, your samp- 
ling and data gathering techniques and your methods of analysis* 

There. a'major problem invol^d when using tjie CIPP Model; It uses a 
very\pomplex methodology. It alsoAtend^ to oyer-value the efficiency of 
educational processes and to undex'value student goals and aims* However, the 
CIPP t(odel^ is one that coyld use either inside staff evaluators or^ outside^ 

educational auditors'. ^ The strength^f the model lies in its co/itinU^d focus 

* ^ 

on evaluation for aecisioo"making.* ^ * ^ 

The CSE Model ' " ^. ■ 

Marvia C» Alkin, Director of the Center of the Study Jof Evaluation at 

UCLA has suggested a rather eclectic evaluation model ' (1970) . The basic 
* 

* 

principle of the CSE Model is that evaluation is a continuing process which 
aids decision-makers to select better among alternatives. (T^e latter element 
seems to be a universal.) Yet, it should be noted that to jelect amSng alter- 
natives means just th^ti Evaluation mtlst be viewed, argues Alkin^ as a means 



by which directions might be changed, programs modified or jeliminated and 

personneL changed as need be. In most cases, vpcational ^evaluators^ do not use 

evaluation to specify alternative directions i The identification of such 

alternatives^ then^ becomes a function of the evaluator^ 

CSE elements . ' All-in (1970) identified five evaluation requirements^ and 
*, " ' * 

their concomitant decision*areas< The five coupletSi if you will, a^e; 



Decisions - H ' 

1-^ Selection of lObjective^ or Problems 

2, .Programs to meet jObjectives 

3. ■ Program Operations 
4* Program^ Improvemdat 

5 * Program ^erti f ication 



Evaluations* 
Needs Assessment 
Plans ^ 
Implementation 
Progr*ess 
Outcome 



It may be noted that /these elements are similaV to many, of those already 
discussed* AXkin stresses that, each of the five couplet areas C coup let is our 
term) requires collecition of informationi an evaluation of that information or 
data, and finally a' division ba&ed ^'on 'the 'in^ffcation. In all steps the 



eyaluator must realize tnat the judgments are based only on ai^ probability of 
success < t , 

Whed one judges a vocational program, the first two element "couplets 
would be most critical, i-e/, oT^j-^ctives-needS assessments and program plans* 

However, tihe judging of instructional components of the vocational program 

... 

would rely chiefly on the last three element-couplets, i.e. , operations- 
implementations, improvement-progr'ess, and certification-outcome. 

V . ^ ' 

One poia^ remains forem^^t in the CSE Model; The emphasis' is always on 
Improved de9i^ion-making capabilitieis of the persons* who are directly affected 
by or who hav6 effect on the vocational 'programs by continuous* data collec- 
tions^ In short, the process is never ending. 



Formative and Summative Evaluation 



A basic objective."%>£ any .evaluation system is to determine the extent to 

, .■ V -' ' . V ■ . ■ * 

Which the projej;t objectives afe being achieved and the impact that the-pro- 
jpct is' having on the p'^artidpaiits* To, accomplish this evaluation^obpective, 
two additional evaluation me^^o\olog;Les are offered for use to vocational 
educators:' (1) formative, and (2) summative* None other than Benjamin 
Bld^m (1968 and 1971) and others have suggested this mode. Let us examine* its 

\ 

components* , , ■ , ^ * " ^ j . 

Fonoative evaluation . Formative evaluation is ^designed to .provide feed- 

. V ■ ■ ' \ r ■■ ;r . . 

back in a rather immediate sense* Fo^rmative. instruments are specifically 
designed 'tt> monitor aspects of any /program determine where problems^ are 

. -* - I . ■ " 

emerging. By using formative evaluations probljsms vill be gfuickly identified 

and rectified* Tor example, £f some inethbdology is being used which causes 

' * 

' - . ■ - ■■ - " 

the students to do poorly ^ it will be through formative evaluation that 

" * , " ' . ^' 

trouble**shooting may take place* Often teachers give assignments but do not 

^ ' V . . ■ ■ r t 

check the students^' work until the conclusion of the project — whi9h is too 

latei* By continually checking the "smalT steps," voc^Bl^al teachers might 

; ' . ^ ' ' ^ ■ ■ ■ 

identify * potentially detrimental learning or even instJ^ctional problems* 
^1 ■ . * . 

Evaluators should initiate formative evaluation techniques so that^ they can 

, ^ ' * * " 

observe *many different perspectives of the. program or course while it is 

: . . - . ■ ■ . ■ 

actually in'ojperation. , / , - < ■ ' , 

' /- . - ... ^ ■ 

Only a few selected items need, to be chetked <it any one formative eval' 

uation* These Would all be based on the sj:ated learning. or project ob'^ec- 

tives* One doesn't need a lengthy listing of items. The importaht point is 

that the feedback is' collected while there is adequate time to make aMjust- 

ments to the detailed project planS* - ^ 

The rationale for^ formative evaluation is to provide data on Ji^hich to 

make cprrectives^*immediateiy, if not sooner! When students and program 



staff members alike realize that they ^e\ being constantly monitored and 
helped^ they tend to become more responsibleland become more productive. The 
instructional climate and total program environment become positive and 



siqjportive. That is precisely the kind of learning climate that one ought to 
always subscribe when directing vocational classes. Conversely, vocational 

I V" 

research project^ have "gone on the rocks" because the project director was 

1 \ ^ 

not ^ evaluating project activities over short periods of time but waited until 
the very end of the project to accomplish a one^shot final evaluation. 



1 evaluation. ^ 



Using formative evaluation is much more subtle than simply specifying 



performance objectives. Formative evaluation -requires t^at the vocational 
project director carefully observe ^a selected ,set of experiences for all 
participants. For^ example, in most yocational programs^ some form of labora- 
tory activity is used to build a dusker of generic skills for future use. A 
person subscribing to 'Normative evaluation would monitor the skills and, when 
a student did poorly^ would provid'3 a new set, Of experiences. To correct any 
noted learning de'fi^iency^ one cannot wait to take the ''final exam." Correc- 
tives aire an integral part of the formative evaluation plan. . ^ v 

For. example; one simple' method by which to record formative data is to 
tabul^^te the absolute numbers or percentagesj of both individual or group 
activities. A project director could compare group data on a graph so that 
the directions of the groups could be visually ^isplryed for instant analysis. 



evaluation is that '*hard data" 
'"objective" evaluation of what 



, The essential characteristic of formative 

ft 

are^beinig collected so tha' they may add a more 
is usually considered as "soft data" or the subjective elements of' evaluation. 
But, more importantly^ corrections are built yito the scheme » so that feedback 
is used when it is needed most — not sf^^e d fc/r the future. 

/ 

Summative ^valuation . That evaluatioiy which is conducted as the finarl or 
concluding task is called summative - evaluatio n . We ^>hould note that^ summative 
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evaluation might be the final formative evaluation of a course or project, 
Sumroative evaluations may take several forms, just so'that they are consistent 
with the prescribed objectives pf * the prog ram , course or module , Again , 
suinmative data could be tabulated into absolute responses and then as a pe,rcen- 
tage for each item. Comparisons between students could also be made on summa^ 
tiveMata (but not on' the formative measures). The final grade is, of course, 
determined by the summative /evaluations — N,B,, plura,l — evaluations . Oije does 
not have one summative ev/iluat^on. These evaluations are placed at logical 

''^ I- 

points in the . program or^ project, such as at the ends^ of units,, chapters, 
modules or learning^ activity packages. 

The summative sets /could then be arranged in a profile to illustrate the 

suiir^?sf "evaluation activities. Formative d^ta would provide feedback; whi,le 

■ ' / ■ 

the summative scores would lead to "grades'* or "judgments'* about the quality 

of the performances. ^ 

Most projects fail because evaluation is a one-shot, post-e valuation* 

\ ' ^ ^ ' - ■ ■ 

Suet;, an evaluation strategy can never aid a project. It's too late, for the 
project xs over. 

\ / 

Of course,\ it may be argued .that formative and summative tedjmiques^will 



cause the direction of the project to change , Why, yes , we agree ; and we 
would submit thai if properly usedj the oojectives of the prl^Ject might even 



be altered. 



/ 



Success is the i/nde^lying tenet of this technique. If a program, course 
or project needs l:p be modified because of unrealistic expectations (objec*; 



tives), then why not\ /alter it?. 



Perhaps, the mi)jt compelling reason in, support of the formative ^nd summa- 
tive fflodel is thati there are really no ''surprises" at the end of the project. 
With early feedba<k systems being built into the system, all elements should 



converge on success 
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' Single S ubjcfct DesiRa Hodel 

Although the eiftphasis in vocational educatiop is usually on the use of 
groups as participants, ther:c has emerged a rather new , evaluation technique, 
commonly c^led "single su!>j6ct design-'* This technique is most closely 
associated with contingency maaagement or behavior modification techniques , 
Its popularity sterns from the rationale that an individual provides bo*"h a 
control and experimental basis, depending on the criterion of intervention* 
Bather simply enplainedj the process contains at least five major steps.^ 

The initial step is to determine some "inappropriate** behavior that the 
subject demonstrates or perhaps determine some behavior that you deem approp* 
riate which the subject lacks* The second step .is to actually count the times 
that the inappropriate behavior or action is shown'by the subject. This is 
Called establishing ^ rate tount or es^blishing the - baseline (niunber of 
responses divided by time). See Figure 3-2 for an example. 

The third step ds to Intervene with some specific act which may cause the 
subject to accelerate or decelerate the initially described behavior or action* 
Rate counts are maintained during all phases. If the new behavior changes in 
tbe -direction desired away from the baseline, then you may , have found the 
correct **reinfor.cer** or intervening variable* 

The unique asp'^ct of this design is not to s'top at step three but to add 
step four — revision back to the conditions as they eVisted initially*^ Such a 
reversal should then cause the subject to revert to the^ original behavior or 
action and to show a rate count approximating the baseline* If one observes 
that the desired direction continues even after the intervening v^j^iable has 
i ^n withdrawn, then learning can be assumed to have taken place with a very 
powerful rtfinforcer or extinguisher. An inference could be made that the 
subject is now, to use a nonbehavioral term, a "self-actualizer." 

Reintroduction of the intervening variable used in step three is often 
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Figure 3-2* Typical Chart Illustrating Student ^'Turning in Work** Behaviors 

During Four, Phases of Behavior Modification Paradigm using Single Subject 
Design* 
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the final experimeritor action. If the appropriate acti(^n or reinforcer has 
been determined} then the subject will once again exhibit behavior and a rate 

r 

count similar to that ^during the third phase of this process* 

using single subject design you ran use small nujitbers of individuals 
to act as intact /^groups^^ for cont:rol and experimental phase&i respectively! 
for your study. This design allows for easy replication or duplication^ which 
in large group projects is, both difficult to accomplish and costly to conduct. 

For more information and numerous reports refer to the Journal of Applied 
Behavioral Analysis which publishes 1 almost exclusively! single subject design 
studies. 

In Conclusion ' ^ \ 

• " ■ 1 

The above may all sound overpowering to the novice who has a "good ideaV 



about evaluation. But remember! ideas are not implemented — the procedures\ 



\ 



are! Collectively! ^he evaluation procedures and learning objectives must 
support each other^ This does not irean that your creativity is being curbed^ 
It simply means that your creative efforts must be logically and systemat- 
icariy^ developed so that the ideas can be evaluated. We speculate that with 
PL 94-142 many more single subject evaluation d^f^igns will emerge in vocation* 
al education. 

Ve have attempted to present a selected yet wide array of evaluaticj] 
(decision-making) models which have relevance for vocational educators. The 
implied emphases are on systematic evaluation model^! rather than a crisis or 
**hit-n-miss*^ evaluation. As vocational di relators or teachers you must make 
the value-judgment as to wViat you^ll evaluate ^and how. The above models are 
somewhat complex! but they will measure or judge selected performances « Yet! 
Julian Stanley (1954) cites a reference to an early evaluation technique which 
occurred in the Bible (King James Version! 12:5-6) taken from the Book of 
Judges . 



And the Gileadites' took the passages of Jordani before the Ephrai- 
mites: and it was so, that when those Ephraimites which were es- 
caped said^ Let me go over; that the men of Gilead sa^'d unto him^ 
Art thou an Epbraimite? If be" said. Nay, then said tliey unto him. 
Say now Shibboleth: and he said Sibboleth: ^ for he could not frame 
to "pronounce it right. Tbeh tbey took him, and slew him at the 
passages of Jordan: and there fell at that^time of the Ephraimites 
forty and two thousand. . / 

« 

It may be noted that the foregoing evjaluation was simple and crude, but it did 

t> 

require one to display the "critical performance." 
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CHAPtER 4 - 
PROCEDURES TO DEVISE AK EVALUATION SYSTEM . 

!I^ere is, a need vhen devising or designing an evaluation system to develop 
a comprehensive perspective within which specific procedures may be conducted* 
This chapter is designed to accomplish that perspective, in part, by assisting 

vocal:ional evaluators in identifying sources of evidence associated with data 

> 

collection and by identifying a number of procedures that might be followed". 
DESIGNING A COMPREHENSIVE EVALUATION SYSTEM 

I 

According to Scarvia 3. Anderson and Samuel Ball (1978), vocational 

educators tend, tp use one of four designs ^when cojrlecting evaluative data: 

ft ' 

(1) surveys , (2) student, or teachers assessments , (3) guasiexperimental 

« 

designs, and (4) experimental designs. As these 'four types, are so widely ^ 
used. Figure 4-1 was prepared to illustrate a matrix^ of selected sources o£^ 
evidence which are available to the vocational educator. To plan an appro- 
priate data^collecting. system there may be additional requirements for addi- 
tional characteristics other than those listed in Figure 4^1, If that is your 
specific case, then the feat to modify is easily accomplished by constructing 
a matrix similar to that in Figure 4-1 but by singly .adding the additional 
problems or variables which liave been identified as locally needed. 

Initial Tasks in Desij^ing an Evaluation System \ 

* 

Vlhen designing an evaluation system for the first time which is more ^ 
coinprehensive than **ad hoc," there are several tasks which need to be done 
prior to collecting any data. The first task is to write the objectives 
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for the evaluation system. After writing the objectives, you should prepare a 

\ 

set of policies and> procedures which are convergent with the stated evaluation 

* \ 

objectives. Finally, you, ^^|iipuld identify the audience for whom the evaluation 
results will be dissemina,|:e<l and used. The initial evaluation design should 
iddntiiy one of the evaluation models which might be appropriate to the local 
situation. For example, we discussed the CIPP, Tylerian and other models in^ 

Chaptjer 3. You may wish to synthesize from some of the, various components of 

1 

the models to generate your own eclectic evaluatiqn model^. 

Usinjj evaluation <Juestions . Once the above have been accomplished , it 
then becomes an easy task to identify the specific evaluation activities to 
which the evaluation system will be addressed. Typically; evaluators prepare 
*: list of written questions to aid in the preparation of the specific evalua^ 
tion objectives- 

The following is a very short list, a sample of the typical kinds of 
questions that v<^ational educators often address when designing an evaluation 
system. - - * . . ^ 

,1. To what extent did graduates in the Automotive- Technjology program 
achieve the course and i^rogram objectives? , \ 

2. To what extent is the training equipment the Electronic program 
obsolete? * ' - i ' 

3. To what extent are the objectives of the Wood Technology program 
convet;gent with those which are needed to work successfully ^n the field? 

4. To what extent do we n^ed to exCend the professional .competence of 
our instructional staff in the Health Care program? \ • 

5. What percent of the vocational education program graduat^es are being 
employed in their related tiTjades , industries and areas in which they 
trained? ' ' ' \ 
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The above are examples of the kinds of evaluation questions that would be 
appropriate to further investigation through an evaluation system. Then^ of 
course^ one would simply collect data to determine the answers to the above 
questions/ In this regard,-^ the matrix as shown in Figure 4-1 could be 
slightly modified so that the 'sources of information to be used would lead to 
the current state-^f-the,-art of those questions* \ ^ ^ ^ * 

Using the table of specifications . Table 4-1 illustrates the use of a 
table of specifications when one uses any of the prototype models for curric- 
ulum evaluation. Note that we identi'fied only five models in the table. 
There are seven different ccupponents identified which would be addressed in 
the table. 'We just completed the specifications for Model 1, ^'School Accred- 
itation." You could complete the grid as necessary for your own particular^ 
evaluation needs. / ^ . - . . 

There are several additional models following Figure 4-1 and Table 47I. 

- - 

These could be used for the tabulation, of specific elements or bits of data 

.that, would help to make^ better judgments about your vocational e^ucatioa 

program. Our rationale for including Tigi^fre 4-1, Table 4-1, and Models ^4-1 

through 4-4* is to provide a graphic portrayal of how the various elements or 

components^ may be displayed* Drscussing what kinds of data ^pugnt to be 

collected is simple. By. showing how the information will be displayed, evai- 

• * 

uators may reconsider their earlier decisions about what information is needed, 
how it will be displayed, and, ultimately,' how it will be used 

All of the illustrations are designed so that they may be easily adapted 
to meet your local situation. We encourage yjou to "cut-and-paste** these 
models 50 that they may be adapted to the specific evaluation system of your 
choice. 
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laves tigative 
Techniques and Problems 



Suggested Sources of Information 



I , * Surveys^ i ■ 

' Opinions about programs and 
Program delivery systems 

Human power needs projections 

Program characteristics 

II. Student/teacher assessment 

Profiles of prospective, 
' entering, and leaving students 

Characteristics of educational 
^taff 

III. Quasi-experimental studied 

Student perfonaance through 
changes of time 

a Student. performance .for ^different 
program components ^ 

Predtdtion oi student success— 
exposed — not exposed to program - 

IV. Experimental studies 

Pifference in performance of 
students in program and students 
not in- program, ^ ^ 
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to program variations 



Different prcygrara effects with 
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ch4):a c te ris ti c s 
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Figure 4-1, Matrix to Determine Evaluative Data Needs 
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TABLE 4-1*- 

TABLE OF. SPECIFICATIONS f6r 
PROTOTYPES OF CURRICULUM EVALUATION 



OUTS I BE 

KEY . .KEY .E;(PERTS 

MOBEL EMPIiASIS' KEY USE ACTIVITIES SEEBEB 



* 


1* School . 
^ Accreditation 


staff^ - 
self-study 


review and 
content 
procedures 
of instruc* 
tion 


discussion'& 
professional ' 
judgments;^ 


none - , - 
unless' . 
other^ 
profes- 
sionals 


c 

2. Tylerian Model 
Ralph W. Tyler 








\ ■ \ " 

: fl 


3. ClfV 
Sahiel 
Stuf flebeara 


\ 




'■ — # — 

• 

■4 


\ 


4, CSE . , ' ■ 
Marvin C. Alkin 


* 




f 

k * 

■ / 


V 

\ 


5: Formative and 
Suiiiniatlve 
^ Benjamin 
" ' Bloom, «t aX 

* 

* 


1 " . ■ 


4 


t 

* 

1 

1 

b 

* 


* 



-Source: Alberta Hill, et al, Curriculum for Graduate Program to Prepare 
Vocational Education Curriculum Specialists . Pullman, WA, Washington. State 
University, DHEW Grant No. DEC 0-74-9287,' 1976. 
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table 4-1, continued 

'tabie of specifications for/ 
protoxypes of curriculum evaluation 



MODEL 



TEACHING STAFF- 
INVOLVEMENT 



/ 



RISKS 



4 . 



APPLICATION ,\ 



1 



!• * School 

Accreditation 


hi^h involveioent 
esseotiali evalu- 
ation based 00 
' staff i&rpact 

i 

\ 


subjective, 

extensive, Y 

time involve-V 

meat oi staff 
i ■ 

t 


involves, staff in 
leadership and ' ' ■ 
program/instruc- ' 
tional evaluatiotl 


2. Tyletiao Model 
Ralph w: Tyler 

\ 
\ 






/ ■• ■ . ■ 


* 


'3\ fIPP ' ■ / ' 
\ Daniel 

Stuffleoeam' . 

Kf 






/ 


^ * t 

f i 

*■ 


4. CSE 

Marvin C. rAlkin 




1 

\. 

E 

1 ^ 

'i 

: / 

/ 


r > 

i 

• 


' / 


5, Formative and 
" Su^mative 
, Benjamin 

BZoom, et al V, 


1 

* 


» ^ 1 . 


i 

\ • 

\ ' » : 



\ ■ 



/ 
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MODEL 4-1 

EVALUATION PROJECT TASK ASSIGNhffiNT 
AHl5 CHECK FORM 



Project 
, Page 



^Date- 



Title 



What needs to 
be done? , 



Who is to 
dfeit? 



To be dqne 
by what date? 



Was it done? 



i 



e^lua^ 



Model 4*1 illustrates An, easy form to , assign staff 



ion tasks I 



This procedure can^ provide positive feedback to the evaluation staff when the 
tasks are completed". On the other hand, using the above form identifies those 
^ho didn't get' their tasks completed during the sp "ified time. Exactly how^ 
this check' form i^s used may vary depending upon the number of person^ involved 
ii^ the investigation, the variety of tasks and sub*tasks, and the ability of 
individuals on the evaluation team do their job independent of supervision. 



GROUP 



MODEL 4-2 

'individual reporting form 
• ^ prtject title 



DATE 



TIMES 



TOTAL * 
HOURS 



WHAT you DID 



WHO VOU 
WORKED WITH 



COMMENTS 



SAT.. 


J ■ 

* 


1 , 

/■» 

% 


r f 

■ t 

* 






SUN. ' 




i 

«^ 


— — — - q 

•< 








7- 




if 




TUBS, 

— 1L 




• * 




t \ 


* 

, V 


WED. 


t 




* ^ % 

3 

1 

' 4 1 


1 ■ 

c 




THUtlS. , 




1 


V 


t 




FItl. 

rj 


* * 






* * 


^ * r 

t 

i 

t 

T 



Model 4-2 illustrates a time reporting form that could be used to collect 
primary data for a project .evaluation where the *tiine spent on a vocational 
' activity was in5>ort4nt to evaluation, ' ^ * 



PROJECT DIRECTOR 

DAY DiVlE srUUCTURED ACTIVITY 



MODEL 4-3 

PROJECT DIRECTORS ACTIVITY EVALUATION 

PROJECT 



LOCATION NATURE OF PARTICIPATION 



roN. 

« ■ 


r 
r 

i 

i 






TOES- 




1 . ' ■ ■, 














\ 


WED. 




! \ - 

! 


P 




TKURS. 






\ 


^ — ' 


FRI- 


- \ 1 

t 

> * 

i 


. A ! 


\ 
i 
I 


i 

4 



Model could be used as aa "ethaogra^hic" grid t6 determine the inter- 

\ ^ : 

actions which take place in a highly personal or interactive i^mject; such i 

E ' 

follow-ups or job placement activities. 
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PROJEQT, 



DESIRED ' 
-INFORMATION 



MODEL 4-4 V 
EVALUATION INFORMATION CHART 



DATA COLLECTION PROCED 



mJ 



TYPE OF 
INSTRUMENT. 



DATES 



PERSON RESPONSIBLE 




I DATA fREATMENTS 



STUDENT DATA. 



TEACHER dAtA 



DIRECTOR / Dr*K)SITlON 



Model 4*4 ^hows how an evaluatot could collect .data from various sources 
and then determine how the data would be disposed^ i.e.^ forwarded for use. 



f * 



Prograiia, Course or Module Evaluative Criteria 

You^ authors have elected to combine the discussion of course and projgram 
evaluation because many secondary and post^secondary vocational programs and 
courses have similar components/ The components that contribute to the effec- 
tiveness of the program or course are the administrative organization, the 
teaching staff , course or orogram objectives^ testing (performance and/or 
written), Content, teaching strategies, students, resources, guidance (career 
counseling, placement and x'^llow-up) - The degree to which each of these 
components is considered may vary according to whether one is addressing one 
course or a comprehensive program. The scope of the evaluation of a program 
or course ought to be based directly on the interrelationships of thft com- 
ponents or course cbaracterist4-cs- , ^ 

For example, Tim L. Wentling and Tom E. Lawson (1975) illustrate the 
interrelationship of course and program evaluations to student ev^iluation. 
They use the following seven tasks to determine - those interrelationships: 

1. Honitor student achievement 

2* Guide student careers 

3* Classify aad place the students in the program 
4*\ Evaluate programs 

5* Improve the instruction and hypothesis testing 
' 6^ Improve the curriculum 

7, Assess teaching effectiveness 

Wentling and lawson also listed ways in which evaluation data can be 
effectively used. For example*, data and information can aid in facilitating 
the vocational program administrators as they talk to parents about students* 
progress and in preparing recommendations for their g^jaHuates' as they seek 
jobs or seek further education* J/hen student progress i^ monitored, voca^^ 
tional educators can review records of their learner's performance to see if 
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certain areas of their instruction Ij^ve been covered* More Importantly, how" 
ever, student ^ progress is monitored to share information and data vith 

Students tcTreceive feedback. Students need* like* and want recognition for 

\ ■ ^ 

^ — \ — 

their achievements/ - - 

' The following example of an evaluation methodology (Model 4-5) is taken 
from a curriculum guide for instructor training of medic laboratory aids. 
Observe how specific terminal behaviors provide, a clear understanding of what 
students should be able to accomplish and what students' should not be expected 
to do* A maximum latitude allows the instructor to develop learning activ- 
ities and various tea ning methods for students. This, in short , is the 
essence of criterion evaluation-- judging the sttident by already existing 
standards* ^ 

Model 4-6 illustrates how a specific performance objeclive can be accom- 
plished on an independent study basis. Evaluation is based on satisfactory 
* 

completion of a grid similar to Table 4-1. 
* ' 

We conclude our models with, another set of specific module or unit objec- 
tives. Model 4-7 is from a federally funded vocational education project." 

Observe how each content outline has an_^€conipanying set of activities 
and resources which allows an individual to move at one's own pace ^ Concom- 
itantly, it requires that all needed study materials and work stations be 
readily available and easily accessable. 



"Vocational Education Curric u lum Spec^^list VECS . "Moduli 7: Derivation 
and Specif icatidn of Instructional Objectives" by American Institutes for 
Research, Palo. Alto, California, 1976. Contract No. OEC-0- 74-9286 from the 
U*S. Office o^ Education under Part I"-Curriculutn Development in Vocational 
and Technical Education, Vocational Education Amendments of 1968, Public I^aw 
90-576, pg, <>.-10t .^.James-A.^Dunny, cd:tor^^ ^ 




MODEL 4-5, STUDENT CRITERIA - 
SUBJECT: LABORATORY SAFETY: Dangerous Substaaces I.D/ Numbet 



RATIONALE: The necessity for proper handling of dangerous substances should be 
carefully emphasized to the laboratory aide^ who may come in contact 
with such substances when ordering or storing supplies ^ preparing 
reagents, or handling specimens. /^^' 

BEHAVIORAL OBJECTIVES: In any real or simulated situation, the laboratory aide 
will be able to describe or demonstrate the proper handling and 
storage of potentially dangerous substances such as acids, bases, ^ 
flammables, mercury, poisons, and biological specimens of any type. 

PREREQUISITE OBJECTIVES: D-3a 



ABILITIES 



RESOURCES 



To achieve the objective, the student should be able. to: 



a. lift and manipulate gallon-size bott^les of caustic 
liquids safely without spillage or breakage, 

b. recognize the sj^nptoms of chemical exposure, 

c. describe the use of antidotes, 

d. recognize the infectious hazards of clinical 
specimens, 

e. recognize specimens of extreme danger, e.g., 
icteric specimens and those labeled **isolation," 

f * ■ identify types of laboratory poisonings, their 
proper labeling and handling, 

describe the unique cjiaracteristics of mercury, 
its hazard ^nd proper handling, 

h. identify hazard labels found in the clinical 
laboratory, 

i. identify laboratory liquids which are flammable, 
describe proper methods of storage and labeling, ^ 

identify stroiW acid and base solutions, describe 
potential for hazardous fumes, proper labeling, 
handling, and ^tora^e, 

report laboratory injuries and accidents to 
supervisor* ^. 



^'-'^ Source: State of Washington. Secondary Curriculum Guide for Medica l 
Laboratory Aids . Superintendent of Public Instruction in cooperation with 
the Commission for Vocational Education^ Olympia; January 1976, pp. 49-^^1* 
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MODEL 4-6; INDEPENDEKT STUDY MODEL 
PERFORMANCE OBJECTIVE . ' 

> 

Explain the key salient features ^ and usage of evaluation models having 
relevance for vocational education. ■ - 

Instructional Objective 

The learner will be able to describe the usage and emphasis of four evaluation 
models . , * 

Learning Activity 

Consult the list of references and complete tbe'**Grid on Prototypes of 
Curriculum Evaluation/* 

References; - - 

Bloomi B,S,j Hastings, J.T,, and Madans, G.H, Handbook on Formative and 
Summative Evaluat ion of Student Learning . New York; McGraw Hill, 
Inc. I 1971;- Chapters 2^-4 and 6, 

- Popham, James W. Educational Evaluation . Englewood Cliffs, N.J.; 
" Prentite Hall Inc, 1975. Chapter 2. 

Taba, Hilda, Teaching Strategies and Cognitive Functioning in Elementary 
School Children . Cooperative Research Project No. 2404, San 
FranciscOi .CA; San Franciscb State College, 1966, 

^ WellmaUi Frank E. "Systems Hodel for Guidance Program Development and 
Evaluation/* Pupil Personnel: A Handbook for Program Development 
and Evaluation , Columbia, MO: University of Mis^fipurii 1971. 



Model 4"6 is adapted from; Alberta Hill^ et al. Curriculum for Graduate 
Program to Prepare Vocational Education Curriculum Specialist ^ "Organizing 
Instructioanl Strategies/' Department, of EducatioUi Washington State 
University, Pullman, WA 99164. Grant No. DEC 0-74-9287, 1976. 
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MODEL 4-7, MODULE EVALUATIVE CRI^RIA 
■ PART II . 
' CONTENT AND STUDY ACTIVITIES 
Goal 7-1 ^ 
CONTENT OUTLINE 



ACTIVITIES-RESOURCES 



Goal 7-1; PerfOrci the Necessary 
Preparatoiry Steps for Systeinatic 
Derivation of Instructional 

Objectives- - ' * ^. 

A* Job Description 

1- Vocational instruction is keyed to 
occupations and jobs- Therefore, 
ihstructional objectives must also 
be derived from jobs. 

2- In order to determine the instructioaal 
objectives that a student should 
achieve to ensure successful' performance 
on the' job, it is fir$t necessary 

to define the job clearly (26)- 

3. A general job description provides the 
basis for a detailed task analysis whic)i, 
in turn, provides the basis for the 
development of instructional materials* 

Job descriptions^ vary in format 
' / format according to the source - 

Howevelr, the usual components are: 

^ . a- location^ anii general working 
conditions J 
b- a general statement of job 'functions 
and relationship; 

J general-duties;- — " 

d^ possible contingeni:^^s£onsi^lit,i&S4_ 
^ e^^ — siini^ly^statea"J^tasks (5), (12). 



B* Task Analysis 



A task analysis is the basis for voca*-, 
tionaJ^^instructioB- The process involves 
starting with the basi- job and breaking 
it down into successively more detailed 
components or levels. The purpose of the 
process is to obtain an adequate 



(26) Individualizing , 
Vocational and Technical 
Instruction , Chap^ 4- 

(5) Instructional 
Systems Development 
for Vocational and 
Technical Training 
pp- 73-74- 

(12) Occupational 
Analysis , Techniques *' 
and Procedures, pp. 33 - 37. 



^See Classroom Activity 
1 in Part IJFI. 



'^See Dispussion Question 
A in Part III. 
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MODEL 4-7, Continued 



CONTENT OUTLIKE (ContO 



ACTIVITJES-RE SOURCES 



definition of ttie job so that effective 
instructional objectives^and learning 
activities can be devised to teach the 
occupation to a willing studentJt* 



^'See Discussion Question 
B in Part III. 



2. ^ The curriculum specialist must be trainied 
to recognize the least detailed level to 
which a task analysis should be taken so 
that effective instruction will result 
and unnecessary task detailing avoided. 
'The curriculum specialist's perception of 
community needs, and his or her estimation 
of the capabilities of the staff to design 
the necessary instruction^ will likely be 
ti^e primary facotrs in other decisions, 
the primary factors in other decisions. 
'factors;[in other decision. 

3. 



(22) Developing 
Vocational Instruction 
Chap. 3, des^zribes a 
a system of task analysis 
designed for the. prac- 
ticing curriculum spe- 
ialist. See also: (26) 
Individualizing Voca- 
tional and Technical 
Instruction ^ Chap. 4, ^ 
T'. See also: (23) 
Procedures for Construc- 
ting and Using T-a^k 
Inventories . Se^y 
also: (12) Occupational 
Analysis Techniques and 



4. 



Many systems exist for p'erforming a task 
analygip-^from single ones to sophist!-* 
cated and extremely complex ones. The 
Th^^ curriculuia specialist will probably 
^ind a system betwt^en the two extremes ^to 
be- most useful (22). 

A first step in performing a task analysis 
is to locate any existing task analyses for 
the occupa^tion under consideration. This 
involves searching publications, indexes ^ 

catalogs, and other references. Possible ^ 
sources ^or .task analyses include (23)^^ 

Source: * Vocational Education Curriculum Specialist VECS . '^Module 7; Derivation 
and Specification of Instructional Objectives'* bj^ American Institutes for 

Research,"^1Palo Alto, ■ California;" 1976; "Contract No . OEC-*0-*74-*9286 from 

the U. S. Office of Edu<^tj^iLunderJpaxt^X^ 

- — - Vocatiobal"^na"TecTmrM Education, Vocational Educational Amendments of 
1968, Public Law 90-576, pp. 9*10. James A, Dunny, editor. 



^(23) Procedu r es for 
Constructing and Using 
Task Inventories , 
Chap.. 2. ^ 
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Prograia Evaluation Review Technique (P£RT) 

One popular method by which planning for^evaluation may be i^nplemented is 
through the Program Evaluation and Review Technique (PERT), PERT is a^jnethod 
for planning* diverse program activities, regardless of how they are coordi-^ 
nated, into manageable processes leading to the project^s successful fruition* 
Note that the emphasis is on management. If a E^T network is established/ 
there is an underlying assumption tKat there will be management concerns about 
planning; organising; motivating; and contlrolling the fiscal,* material and 
human iresources so that their total interactions will be evaluated by a pre- 
determined set of objectives. 

^he PERT system is an atten5>t to facilitate three common dimensJions of " 
managing the project^^-time, costs (or resources), and performances. Once a 
generally agreed upon set of evaluation goals has been identified, all ^oals 
and processes are sub-divided into very specific components and placed on a 
"work division striicture." A work division structure identifies all of the 
components of every major unit. After major units, have been identified, a 
network would then be prepared. Each network is con^osed of events and 
activities* Events are defi'ned as those items which represent the start' or 
completion of an activity but do not consume time, personnel, or resources* 
Activities, are those tasks or jobs which require the utilization of personnel 
and resources over a period of time. The PERT ne^wgrk^^then^is-^eveltyped^^ 



that a .timeline, management check, and cost analysis estimate, as well as 

output products, may b^ identified. . 

An' outstanding source for PERT techniques is available from the United 

States Government Printing Office* That source is: 

Desmond L , Cook, Program Evaluation and Review Technique Applica - 
tions xa Education , U.S. Department of Health, Education, fi.Welfare, 
Office of Education, OE-12024, Cooperative Research Monograph #17, 
Superintendent's of Documents Catalog /jEFS-5. 212:12024, U.S. Govern- 
ment Printing Office, Washington, D.C/, 1966, paperback, 100 pp. 
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Desmond Cookfs* monograph presents ia detail the application of PERT techniques 
to several different educational processes and serves as an invaluable 
resource* ^ ^ 

Figures 4-2 and 4-3, which follow, illustrate a sample of PERT networks 
as v.they would be constructed to aid in the conduct of a survey., e*g. , to 
follow-up graduates or the designing^of a curriculum project* 

The figures show the organization of the major elements which would be 
'predetermined in the initial planning stage* The activities and events are^ 
then arranged in a logical and sequential order* The actual PERT chart shows 
the major' elements. Further, the PERT chart shows the interrelationship of 
each element. ^ - . ^ 

A PERT chart is constructed in several ways. We have found through 
experience that there ar^ at least three major steps; (1) identifying ele- 
ments, (2) clustering them and (3) sequencing them into the network. 

Once you determine what you want to do (your objectives^ and how youUl 
accomplish the project (procedures), you, are then ready for step number one-- 
identifying the elements or tasks- S~modifie(l "brain storm" technique is one 

easy way to accomplish this step^ You simply identify every possible task or 

*- V ^ 

event that you think must be done^. ^ 



Step tWQ_b£flins->vhenr-thesg^asks or events are clustered about some gen- 



eral theme or concept* These logical orderings help to identify major clu^ 
ters of related events on activities* 

In the final step; s^iquencing, the clusters of^ events are combined to 
illustrate a methodical and linear arrangement of tasks ^ events, and activi* 
ties from the beginning to end* . ^ 

For example, ^.n Figure 4-2 activities 3, 4, 5->- 6 and 8 may all be ini- 
tiated autonomously. None^of these activities is totally dependent on any of 
the others* With such planning knowledge j a project director might wish to 
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assign ^specific responsibilities to various teams, The team^ can develop 
reports which can be coordinated at event number 7. Howevfer, it vould not be 
possible to begin event number 12 prior to completion of both events number 7 
and IT. ' 

By establishing a detailed PERT chart or network for ea^.^* major element 
in which change isj being planned i a vist^al schematic is made availablb of 

anticipated activit[ies. PERT networks must always be considered as "best 

1 ' ^ ' - 

predictions,** The interrelationships of all elements become known. Precision 
is then added to the project calendar by determining time and cost estimates 
on the chart. More inipoirtantlyi critical areas are identified in- advance of- 
the event. Program directors can predict and prepare for anticipated contin-* 
gencies or outcomes, Howeveri as a project progresses, there will certainly 
be' unantii:lpated events that require modification of the original networks 
Thusi there ought to be a continuous monitoring of project activities and 
personnel so that the PERT network resembles* a reasonably accurate blueprint 
for action, Finallyi a nipre efficient use of mateiial and human resources 

f _ '4 

will be the outcome of using a PERT network to describe the project. 

One^_C0Uld' even PERT*a seg^nent'^of an evaluation paradigm, for example, 
preparing'a PERT nefiS|ark on the develo^ment^of a district vocational aclji^^ve- 

ment test, Ther^ woulooe at least nine sets of tasks to be sub-divided into 

( , ^ . 

specific activities and then assembled logically into the network* Th^ tasks 

would be to; (1) establish the test's purpose, «(^X-einploy item developers, 

(3) try out an initial form of the test, (4) revise the test, (5) norm th$ 

test, (6) prepare a test manual, (7) administer- the, test, (8) collect and 

analyze^ the data, and (9) prepare a final report about the results* 
4 * , 

'PERT and planning make for luccessful projects. Now on to figures 4-2 
and 4-3. • ' '' * * 



?ro{fcect^ Start 
Jb^ectlve JStart 
Start .Philosophical Evaluation 
Start Psychological Evaluation 
Stiart Content Evaluation 
Start Measurement Evaluation 
Objiectlves Complete 
St^t Structure 




Ion 

■N 



Start 
Stak 

\ 



Determination^ 
Evaluation 



Event Identification 



11^ r Structure Complete * - 21. 

.12. /Start-Instructional Materials 22. 

13. Start Teacher Manti^l ; 23. 

14. Start Student Material ' 24. 

15. Start Audiovisual Aids ^ 25. 
16;^ Start -Evaluation - 26. 

17. Start Reference ^ 27. 

18. Start Dissemination Procedure 

19. ; Start School Sample 28. 

20. ' Start Material Distribution 29. 



Start '^ryout ^ ■ 
Start Evaluation 
Start Final Materials 
Complete Final Materials 
Start Publications 
Start Teacher -Orientation 
Start Administration 

Orientation 
Start Lay Orientation^ 
Project Coniplete 



C. 



FIGURE 4^2. SUMMARY NETWORK FOR CURRICULUM PROJECT' 
^ . ■ 

Source: Desmond L* Cook« Program ^valuation and Review Technique Application In Education (Washlng£6n». 

: USGPO^ U.S. Department of HEW» Office o£^ Education^ Cooperative Research Monograph No. 17) 1966^ pp. 53» 

55. i^iibllc Domain Document. . ' • . > ' - 



Si 



©— <I 




Event Identification 



1* ^'^rt Project 

2* ^Con^lete Objective 

3* Complete^ Data Paradigm 

4-?^ Complete Hypotheses 

5* Start Item Construction 

6* Start Universe Definition 

7. Start Sampling 

8* Stare Sample Selection 



9* Start Tryout 17. 

10. -^tart Final Form 18* 

1\ Start Interviewer Selection 19* 

li£* Complete Administrative Procedures 20, 

l3* Complete Schedules 21* 

14* Start Field Interview 22* 

15* Start Data Coding 23* 

16* Complete Foxlow-up 24* 



Start Tabulation 
Start^ Statlst-tca^ Tests 
Complete Tests, 
Complete Interpretation 
Compl^etb Tables ! 
Complete Charts -' 
Start Narrative ' I 
Complete itarratlve j 



flGmZ 4-:^ SUMMARY HEx*yORK FOR JURJ/EY> RESEARCH PROJECT 



Source: Desmond Cook* Program Evaluation an d R er^rlwevr Technique: Applications In Education ^ USD/HEW i 
Office of Edtxatlon, Cooperative Repc-irch Monograph . i7> U.S. GPO, Wasnlngion, D.C: 1966. Sources, vj 
Figures '^O* 31 and 32, pp* 41-43. tufallc Domain Document. j ' 



Horal'Ethical Considerations 

While it may. not be in vogue to discuss the morality and ethics involved 
in evaluation! we would be remissi nayi n'^gligent if we didn't address the 
topic, at least in passing. 

Henry M. Brickell (1976) p^ublisbed a rather shocking indictment of those 
being evaluat^ed who attempt to influence the evdluators to distort the find- 
ings and subsequent reports. Brickell summed the various pressures he has 
faced as ''external political factors/* In the last analysis, we interpret 
Brickell as subtly suggesting "situation ethics" yhen being an evaluator. 
They are hieing you to do the job so dori*t probe tao deeply. 

The - authors of this monograph h^ve had widespread project evaluation 
experiences! and in at least two cases one of the authors had the rather 
difficult task of requesting that the respective projects be closed down. 
Aad| the political realities wejre not withstanding! 

If an evaluator allows "external political forces'* to influence the 
evaluation there are really only two decisions: (1) write the report objec- 
tively and let the "fur fly" or (2) resign from the evaluation team. The 
taKpayersi by and large, are supporting the projects. They deserve to jiave 
the stewardship of the public domain to be protected from ui;ethical profiters 
and charletons. But, let us address the topic more fully. 

On being ethical ^ It must be recognized that evaluation^ and evaluators 
tend to be "value-free.'* This means that preconceived judgments are not made, 
and that one attempts to collect data by which to make judgments. Yet, the 
judgjnents or evaluations a^e guided by a set of criteria. A trait is judged 
to be poor or out'^tanding by applying those criteria * ^The values of the 
evaluator might even be opposed to the values of the project. Yet, if the 
project has a set of criteria by which it is to be judged^ then the evaluator 
has no other alternative than to u^ those criteria. 

.. - • 8.- 



It must be understot>d> however, that the ^valuator in such cases may file 
an addendum to the evaluation report ^hich criticizes the criteria and may 
even- offer a set of critf^ria- which would be more germane to the objective. 

Judgments are made by evaluatcrs — that's what the process is all about. 
Yet, an evaluator ^an provide alternatives and suggestions that go beyond the 
specific f project.";, to help improve it. Evaluation is for decision-^making. 
Thus^ it behooves, an evaluator to provide a wide array of suggestions^ %trat^ 
egies for improvements and^ if possible, predicted outcomes. 

Finally, the organization or group being evaluated has the option of 
either accepting or rejecting the "advice." 

An evaluator must view any evaluation as a task—^neither as a personal 
confrontation, nor a personal favor. 



IN FINAL CONCLUSION 

Collectively, we have addressed a few concerns about evaluation. We have 
only presented a sample of the many different systems and mo'iels. It is 
essential to realize tha^ al^ evaluation models arise from preselected cri* 
teria. To be su^* , the criteria are subjective, but being subjective should 
not be confused with being arbitrary or dogmatic. Being subjective implies 
that there is a rationale which supports and generates the critisria. Several 
different evaluation methodologies — have bee*\ presented for your use. Each 
has a distinct set of criteria"^ 

The major goal of this monograph is to provide busy vocational educators 
with models which may be either adopted directly or easily ^dapted ^or use. 
The ultimate import of our work is to iiiq[>rove the evaluation of Vocational 
tiducationd at all levels. 
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- . INTRODUCTION TO THE APPE^rDICES . 

ip - 

Selected Formates Which Illustrate 
Evaluation Devices 

A nuniber of evaluative areas and models in vocational education were 
introduced or implied in the previous chapters. The writers of this guide 



have compiled selected sample instruments^ e,g,» check lists^ survey foicms^ 

attitude' scales » interview scheduleii and asso'^rted evaluation forms. These 

examples may be useful to the readers to assist in constructing .their own 

Aaterials for evaluation purposes. The writers haVe intended to provide a 

wide variety of evaluat-ion formats that cut across all fields of vocational 

« education: However^ many of these materials can i)e adapted for specific us& 

in an infinite number of situations, 
* ^ * * 
To assist the reader ^ the selected sample formats represent the following 

ca^egorie^ or program evaluations; student assessment » teacher a'is^ssment 

follow**up« en^loyee surveys and the general category of evaluation. 

Each sample may not be presented in its entirety; how^ever^ references will 

allow the reader to gain access to the remaining materials from the^ listed 

sources^ It is also hoped that the local vocational educators will ^e encour^ 

aged by thes^ materials to thinK creatively and to adapt the material to their 

« 

own fields. What matters primarily is not the specific content of the eval* 
uatioA form but the approach and the point of view* 
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APPENBK A 



IDEA SURVEY FOBM— STIJDENT REACTIONS 'TO INSTRUCTION AND COURSES ^ 

ForiD C 

The f^ol-ldwing teach&r evaluation form is being developed «to evaluate 
, vocational'^technical courses by the Center for Facility Evaluation and Develop- 
ment in Higher Education at Kansas; State University, anticipated that 
it w-tai be available on a fee for service basis in the Fall of 1979. The 
approach used by IDEA focuses upon the improvement of instruction. IIEA con- 
siders the intructor*s course, objectives as well as the size of the class and 
the students' desire to take the course. When students do not report satis- 
factory progress on the instructor's course objectives, teaching methods that 
might be improved are suggested. The items contained in this form were 
selected by over 100 instructors of vocational-technical course^ at over 15 
community colleges and area vocational- technical schools as being appropriate 

i 

for their courses. , * . . 

Appendix A is printed with the permission of the Center, The present 

form C is a research version and is being copyrighted. Anyone wishing to use 

this form must contact;, 

^ Dr, William E. Cashin 

Center for Faculty Evaluation md 

Development in Higher Education 
Kansas State University 
■ 1623 Anderson Avenue 

" . Manhattan, KS 66502 
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IDEA SURVEY l:,ORM"STt>OENT REACTIONS TO INSTRUCTION AND COURSES 

FORM C 



To the Learner: Because this !s part of a research study, the form has more 
questions than the final form will have. Please answer each question CARE^ 
FULLY. CHEQK that the number you are marking on the response card Is the 
SAME as the number of the question you are answering. . 

Your HOWEST answers to the§e questions will be helpful to your instructor 
who will receive back all of the re^sponse cards. 

How often did your instructor do the things described 
below? Use the following code; 

1 - Hardly. Ever 3 = Sometimes 4 = Frequently ' 
Z = Occasionally 5 = Almost Always 

The Instructor: ^ 

'1* Had prepared everything necessary for each class. 

2. Let learners know what they were expected to learn. 

3. Presented coQrse content clearly. 

4. Olstinguishecj between facts and opinions* 

5. Used examples from rea'-life or actual wprk. 

6* ^Summarized in a way which helped learners remember. 
7' Demonstrated the skiMs or procedures learners were required 
to learn* " ^ - 

. 8. Gave learners opportunity to practice skills or' pro^cedures* 
9. ' Taught skills or procedures step-by-step. 
- 10. Related what was being learned to on-the-job situations. 

11* Fit the instruction to the differences in learners' knowledge or 
skill. ' 

12. Used films, models, or ott^r teaching aids that helped learners 
iearn* 

13. Used special teaching approaches (e;g., case.studies, role play- 
ing, contract learning, etc*)* 

14. Provided help when the learners needed it*' 

15* Encouraged real discussions with learners (instead of just asking 
or answerihg questions)* 

16. Found ways to help learners answer their own questions. 

17. Changed way of teaching if learners' were not learning. 
18: Gave learners reasons foe their grades on tests, projects and 

other M'ork* 

19* Used up-to-date materials and equipment. 

20. » Was patient with learners. 

21. Seemed to know when learner was having difficulty* 

22. Respected the learners. 

23. Showed an interest in the goals of the learners. 
24* Was enthusiastic about what was being taught. ' y 
25* Made sure that learners understood directions, safety and health 

considerations for each task* 
26. « Made presentations which were interesting and held learners' 

attencion* ' 
27* Encouraged learners to find v?ays to improve their work. 
^ 28* Praised constructive criticisms of learners' performance* 
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29; Gave constructive criticisms of learners* performance. 

30. Suggested what learners coutd'do to Improve, * 

31. Promptjy returned learners' work- tb them. 

32. /Accurately evaluated the learners* performance. 
^ 33, Enforced the rules and regulations fairly. 

3A^* Gave each learner a fair share of attention. 

35. Pointed out common problems to learners before they began a 
task, * , * ^ 

36. Provided active woFk experiences (e.g.^ hands-on) for the 
learners. - . ^ _ - 

37. Used community resources^ fietd trips^ *etc. 

33. Taught in a well organized way. 

39. Encouraged learners to speak freely and openly. * 

Sf<IP the spaces A through G on the response card. Start with question 
NUMBER 40. ' ' 

On each of the possible objectives listed belo^^ rate the progress 
you have made in this course or program compared with that you 
have made in other courses jor^^regrams you* havTTatteiT^f tRTs" 
>i schoo:. If thiii is the only course or- program you have taken here^ , 

compare your progress with that you have made in previous learning 
experiences* ' Ypu are not^expected to make progression every 
objective in every course or program. 

In this course or program^ my progress was: 

1 = Low (lowest 10% of the courses or programs \ have taken) 

2 ^ Low Average (next 20% of the courses or programs) 

3 ^ Average (middle 40% of the courses or programs) 

4 = ^Igh Average (next 20% of the courses or programs) 

5 = High (highest 10% of th^ courses or programs) 

Progress on: 

40* Learning factual (<nowtedge^ terms and concepts of the occupation* 
41. Understanding the principles or reasons for various practices^r 
procedures. • 
* 42. Knowing when and how to use different methods or procedures. 
43. Correctly using the instruments and tools of the trade or 

occupation* ' . ' 

4^. , Developing the manual skills required in the occupation. 
' 45. Learning to solve problems (di&ghose^ trouble-shoot^ debug^ etc.) 

46. Developing the communications skills (reading^ speaking^ writing) 
needed for the occupation. 

47. Developing the work habits (attendance^ drers^ safet^/ etc.) 
^ needed for the occupation. " ' 

48. 'Learning to nieet the production demands of the occupation. 
49.. Learning the skitfs necessary to get a job (applying^ preparing 

^a resume^ being Interviewed^ etc.)* ' ^ 

50. ^ Developing the attitudes desired of people in the pccupation. 

51. Learning to work with pther people (e,g.^ co-workers^ super- 
visors^ etc. ) * - ^ I 

52. , Developing a sense of personal responsibility^ (self-confidence^ 

self-dlsclpllne. *etc. ). 
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53, Understanding mys^lt/ my strengths and weaknesses related to 
the field or occupation. 

On the next six questions/ compare this course or program with 
others at this/ institution (on^wtth-yojjr previous learning exper* 
iences)/'using the following code: 

' 1= Much less^than most courses or programs 

2 = Less than most ^ ' * 
- 3 = About average 

4^= More than most ^ * ^ " 

5 - Much more than most 

The Course: 

.54, ■ Amount of reading, 

55, Amount of work ia other (non-reading) assignments, 
'''55, Difficulty of subject matter, 

57, How ".well tue course- Ht together (various topics and activities 
were^relat^* to each other), 

58, Adequacy of physical and other facilities (space, lighting, 
equtpmeji't, tools, etc). 

Describe your attitudes toward and. behavior in this course 
or program, using the following code: 

^ ' 1 ^ Definitely false - ' , 

2 ~ IVtore false than true* * 

3 = In between ^ - - 
/ 4 ^•More true than false 

5 = Definitely true ' * 

, ^ Self-rating: 

< , ^9, I worked harder on this course than on most courses I have 
taken. - 

60, , Before enrolling, i really wanted to take this course, 
51, I would like to take another course from this instructor, 
62, Because of taking this- course, I like this occupation better. 

Your comments are invited, on how the instructor might improve this course or 
teaching procedures. Use the back of the Response Card. 



THANK YOU FOR YOUR 'RELP, 
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APPENDIX B 




SAFETY EyALUATIOK 



Safety* is a ^'major concern to ^anyone * involved in vocational program 
operation* Tjie set of criteria identified in the 'selected form can be 

easily applied to a variety of fields or di£l5piplines of Vjocational Education^ 

' . ^ ^ 

Source; Oregon State 0epartn)^nt of Education < 

-<;oinmunity College jS_£areer Education^ ' " 

* Teohnicai & Industrial Education, '"Metals Cluster, Curriculum" 
February, 1974- This form" has been modified to more readily 
meet the needs of evaluators in all vocational programs. 

Directions 

The following" .sample instrument may be used by local vocational directors 
or teachers^ to assess the adequacy of th^ir safety prpgram. The scale^,^^ 4 3 2 
1 0 if consistantly used can provide your vocational education staff with use- 
ful information across* programs to^ establish safety standards C^ithin a given 
facility, A "4" would indicate the highest possible ranking for any item, 
while a "0" would indicate a most Unsatisfactory rating. 
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/ " elemeUt safety 



/ .Safety education is an integral paft^Q£_i:he vocational programs. All instruct 
tional pex^oonel :ihould be knowledgeable in the application of all Washington 
safety codes related to' the vocational program. In order^ to insure safe 
working conditions^ it is imperative that an adequate shop or lab safety 
program be established ^ including periodic inspections, involving, staffs ' 

' students and-qu^lfied safety inspectors. The school district should accept 
the ^responsibility to organize '\nd maintain a total school ysalfety^ program and 
prov:^de the. Iiecessary administrative le^adership} in-^service^ and required 
safety materials and equipmeftt^ \ \ \ ^ 

~ ^ ^ ^ ^ \ ~~ 

ASSESSMEMT CRITERIA : . ^ . * ^ ASSESSMENT REMARK . 

1,' There is a functibning/shopjafety 4 3 2 10 ' / 

* , program 'including periodic inspec- ' / , 

tions involving staff students and ' ' % 

" qualified safety ihspe,ctors. * 

^ 2- In-service is provided ^t regular 4 3 2 1 0' - ;r 

intervals to insure that all ^ . / . 

instructors are kept up date - . • 
in all safety practices and {Policies* ^ . . 

^^^3- Adequate funding is available to * 4 3 2 10 / 

provide the required materials and ■ / 

equipment to maintain a^safe pr ogram 

* . ■■ T 

* There is an established policy 4 3 2 10 

-covering firsi'^ai'd procedures. 



/ 



/ 
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5* The shop program is an integral part. 4 3 2 10 
of the total school safety program. 

6» 'School personnel are designated as . 4 3 2 1 0 
safety officers with the specific 
responsibility to adminljstrate the 
safety. progra^f maintai^PMl required >. 
rec6rds^j and. report all violations* 

7* ^ The shop has an adequateNrentilation ^-4 3 2 1 0,'^ 

stystem for the specifi<; .wca tional * - 

'program*' ^ * * - . 

8. Students are provided with eye '* " — 4 3 2 1 0 ' / 
protection equipment ^ and machines . ^ ^ \ / ^ 

' are guar^d properly* * . . | ' 

9. *^ The arrangement of the shop allowfi^ 4 3 2 1 0 
* for a safe traffic pattern^ as well 

as /safety zones around machinery ' ' " ^ 

and equipment . " ' . ^ 

AVERAGE TOTAL ASSESSMENT 
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/ ' APPENDIX C : , 

. YOUtH ORGANIZATION BV^UATION 

«o5t -educator, .mli ag^- that youth organizatii^ ari^ an integral' part ■ 
of the educational progra,. Much of the developing Pfrcptual ahilifie. of 
.tud.hb and af'flr^ation of attitude take, place «her^_ student, affect ^he 
governance if th^.elves. The following in^tru^ent i. provided to a..e.. the 
yluth organization, a.pect.of the vocational education progran,. ' _ 

^ Source: Oregon Department of Education, - ■ * 
^ "Assessnint Manual for Program Planning .' 

Model Electrical Cluster Guide, \ 
May 31, 11975. ^ ^ _ , 

• The fo'llowiig sampl. instrument be used to assess ti.e youth organiza^ 

^ t^ons prog- wUer,:VICA/m, FHA, FBu'or DECA. The.percents for each^ 

item in the*per|ormani ..tand,rds section of the instr^nt is^ used to estah- 
' lish a norm for^he enUre" iVstru^ent. These maxima percents are suggested 
' amounts based , on- the experience of educators worRing extensively with youth 

groups. 'xhevocLonal director or teacher evaluating a youth program can use , 
' the model range cale to determine how well their youth programs are achieving 
. the performance standards. . 
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YOUTH ORGANIZATION 



1 



youth developmeEt should be an integral part of t|[ie vocational education 
program. This should includes * ■ 

Personal growth-rindividual self'-iinprovement through scholarship, 
xitizenshipi and |)articipation^ in homei school /and community^ 
activities.* ^Z*-/ , - ^ - " 

Community uiiderstanding--impravement of , relations among students i 
^ b^stf/een students aifd €ea<:hers^ employees, and. employersi managc:nient 

and labor, sch*>ol and community and. other nations. y 

, Safety-'in shops> c^$sEoomS| on the job, and on the highway, \ 

Teacher recruitment-*Jthe ^encouragement of capable students 'to enter 



the field of industrial education* 



i 



Vocational youth cooperation-'-among youth, in all areas of vocational 
^education. ! 



Good public relations-'-'promoting a general public' awareness of the 
good work that youths engaged in vocational education are doi„g to 
$\ better themselves, and their community; state^^nation and world. 

PERFORMANCE STAIflJARDS ASSESSMENT 



1. 



3. 



4. 
A 
5; 



Is there a VICA Chapter in. your 
school and how are students in-* 
formed of its existence? (^15%) 

Are majority of the . , ^ 

^ students in VICA? (40%) 

' / ; - ^ ^ ^ 

Do students in local VICX Chapter 
'participate in state and national 
VICAevents (10%) 

Do VICA in?mbers pa'rticipate is local " 
business and community affairs? (10%) 

Does ^he srhiCRdr^dministration 
support. VICA" involvement through 
teacher release' time and a minimum 
amount! of financial, assistance? (15%^) 

/ 

Is the involvement in VICA an integral 
part of the\vocational' offering? (10%) 



ASSESSMENT SCALE 



20 



40 60 



Model^Range 
80 *^ 100 



APPENDIX D 
GUIDANCE PROGRAM EVALUATION 



1^ 



\ 

'Evaluation of a Icareer and vocational guidance program is essential to 

vocational education, The authors have included these criteria in the follow- 

ing format so that they may be modified to fit local needs. 

Source : "Health Occupations Cluster Program Assessment Instrument for 

^ Secondary Schools", Oregon Department of Education. December* 1977, 

.The following sample instrument may be used to assess. guidance and coun- 
seling program a£ the local level, ; The percents for each item in the perform- 
ance standard section of the instrument is used to establish a norm for the 



entire instrumenY^ These" maximum percents are given as suggested maxima based 

« 

on experience of educal^brs working extensively with career-guidance. 

The vocational director who evaluates the guidance and counseling program 
^can use the model range scale to determine how well the school district's 
guidance and counseling program is meeting the needs of students. 
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EI/EMENTS 



PERFORMANCE STANDARDS 



ELEMENT: GUIDANCE, COUNSElING> FOR VOCATJONAL aUSTERS 

A model vocational cluster should be structured so that 
assistaace caa be given e^tn student in the achievement 
of the goals and objecjiives consistent with interests and 
abilities. 

PERFORMANCE OBJECTIVES: . 



The vocational cluf^^r faculty 

in cooperation with the guidance and 

counseling dej^rtmeht has: ' y 

1, ^ Provided occupational and career 
ittf-^Wation which will enable 
^J^udents to assess ^their l.nterests 
..^and aptitudes. /"^ 



2. Developed and implementeil^a plan 
for/Student enrollment "which will 
include: community needs ^ student 
interests » and occupational require- 
'ments as factors foi; consideration. 

,il ! .. ^ 

Encouraged students to develop a 
^total career program by tajting 
supporting courses appropriate to 
their career interes.ts* 



4.. Provided facts concerning post- ^ 
secondary educational programs 
and the competitive selection 
pj;ocesses for those programs** 
t 

De^^eloped and implemented a program 
to provide exploratory courses in 
the occupations for students in 
grades 7 through 10* 

6. Developed a plan to make students ^ 
in grades K-6 aware of the oppor- 
tunities and rewards in business 
and industry. 



Are counseling/ guidance and 
other staff members responsi- 
'^ble for advising students in 
the selection of -courses and 
programs , informed about the 
vocational career areas and 
appropriate allied support-, 
ing courses? ' (20%) ^ 

Are students aware of the com- 
petitive ^nature of selection 
procedures in post-secondary 
education for specified voca- 
tional program? (20%) ^ 



Are students referred to 
Vocational Cluster as a resulJL, 
of a demonstrated interest an^S 
the ability to meet occupa- 
tional requirements? 
(20%) ' 

Is Student enrollment b^sed^ on 
3 plan that is justifiable in 
terms of a* community needs? 
C2p%) ' ^ 

Are exploratory experiences 
available in vocational educa- 
tion for students in grades 
7-10? (10%) 

Does a plan exist for voca- 
tional orientation fox^ stu- 
dents K-6? (10%) . 



^SSESSMENl' SCALE 
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APPENDIX E 



"PERMISSION TO BEi RODUCE THIS 
MATERIAL HAS BEEN GBAMTED BY 



Johr) R^i\Af_ 



TO THE EDUCATIONAL BESOUBCES 
INFORMATION CENTER {ERIC)/* 



JOB PROGRESS EVALUATION 



The final stages of most vocational education training programs have a 
cooperative work experience component. The success of this phase can be best 
measured if the employer^ the supervisor and student .are given opportunities 
to provide input. ^ ^ 

* \ 

The following sample forms are provided for the reader to identify 

appropriate content and sample format. 

These samples were selected at random. 

Source: Vocational Teacher Education Department 
University of Idaho 
Moscow, Idaho> 1976. , 
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CRITERIA FOR SEBECIION OF A 
TRAINING STATION 



if 1 . * 

training station ^should be selected pa the needs of the Sc ooperative 
s^ their career objectives^ the community akedk ^j^nd thm standards 



A 

students 

of the emplbyer* 




2 points ~ tc a great extent 

1 |»oint = to a limited extent C 



r 

p points ='does not offer 



To what Extend does the training station offer: 

1* interested and understanding employers 

2* progressive attitude 

3. safe arfd sanitary work conditions 
4* good image - 

— 5- convenient-location .~ 

' ' " * 

6* varied learning e:qperience 

7.^ opportunity for advancement and promotion - 

, 8* provide a minimum of 540 working hours per year 

9. wages paid to other employees; in the same position 

TOTAL 



i 
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COOPERATIVE EDUCATION 
' JOB PROGRESS REPORT 

HIGH SCHOOL 



TO THE EMPLOYER: This report is your estimate of this student-ti^aineeVg^ work 
activity. Please check the chart in the columns which most accurately 
describe this particular, student. 



Date 



Cooperative Education Coordinator 



— St^en^tr-Empioyee-*-s^fame^- 
Firm; 

Job Assignment: 



y 



Failure 



Needs 



Above 



APPEARANCE: 
' Proper Clothes for Job 
WellMroomed 
Personal 'Cleanliness ^ 

PERSONALITY: 

Willingness to Learn 

Enthusiasm 

Tactful 

Well Hannered 

DEPEJfl3AB]ixT?.v^ 
Regular Attendance 
Punctuality 
Respects Rules 
Completion of^ 
Assigned Tasks 

ATTITUDE: 
Toward Job 
Toward Criticism 
Toward Other Employees 
Toward Customers 

KNOWLEDGE OF JOB: 
Job Procedures 
Selling Hethods • 
Supervision, Required 
Accuracy 

Follows Instructions 



/ 







V 
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Point t^o be Rated 



Failure 



Needs 



Above 



Exceptional 



J--* 

WOM ACCfiMPtlSHED: 
Indu^^ious * 
Uses Initiative 
Take^ Pride 
Qu'%0.ty of Work 
Quantity^of^ Work 




1 

















































Signed: 
Title: 



You may use the reverse side of this shelst for your additional' or explanatory 
remarks . * t 



APPENDIX F 
TEST UEVELOPMENT INSTRUMENT GUIDE 



"PERMISSION TO REPRODtJCE THIS 
MATERIAL HAS BEEN GflANTEO BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATJON CENTER (ERICJ." 



0 



The following excerpts are from a guide devetoped by Educational Testing 
Service that is intended to help in the creation of well constructed multiple 
choice test itenis. 

The elements included in this document are test item terminologies and a 

* * 

checklist used to evaluat the overall quality of individual multiple choice 
Items. - ' ^ 

The test development guide can help teachers with minimal test construe;* 
tion training .and experience to improve their skill in test contruction, and 
it will assist them to develop more reliable feedback* about how well they are 
teaching and how much their students know. 

Appendix F* is printed with" written permission from the Educational Testing 



Service (ETS), Princeton, NJ 08541. Anyone . desiring to duplicate pages 
or J^tf must obtain written permission frofn^ETS, \ 
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Mtfltiple choice test question terminology , 

Mulhple choice item : a test; question in which a number of alternative 

response choices are given from which the correct answer is'to be 
I ■ ■* 

' r f 

selected. Most such items use 4 or 5 choices^ usually identified by the 

letters A-E. ^ 

Discrete item : a s|?ngle multiple-choice question, including the cliolces* 

Stem : "^he initial ^art of the item in^^Sct the task is delineated it 

may be a question, directions^ 'or. a]> incoqrplete statement. 

Options : all the choj^es in an item. 

Kejr: the correct answer. 

Distractors : / the incorrect options 



Illustration: . ■ * . " * 

Discrete *Ite m 

Which of th^ following was the. primary ^turpose for which 
craft guilds were formed in the Middle Ages? 



; Stem 



(A) The training of new workmen, 

(B) The distribution 



(C) The sociaj and political ' 
advance of their members.... 

(D) The control of town govts. j 

(E) The regulation of production.— 



Distractors 



tions 



Item Sets : two or more items based on a common passage* problem* graph 
experiment* chart* or other stimulus materials. 



Please use the following as a review checklist after completion of your test 
item. Each of the following statements should be checked ( ) upon completion. 
If any of the statements do not characterize your test item, it should be 
revised to meet the checklist criteria. 

1. Does the stem present a single definite problem? 



Soes the problem presented in the stem adequately test the 
performance objective? 

_3. Is tK^e problem presented clearly and simply? The item should not^be 
a test of reading ability* 

Is all the information present ^n the stem *Ln order for the examinee 
to understand the intent of the item?' 

5/ Are the options presented clearly and simply? 
_6. Are all the repetiticJus wordings removed from the options?. 
J . " Are the options grammatically correct as completions of the ^tem? 

8. Are the options written in paralled form? 

_9* \ATe the distractors properly worded so that they are not too * 
technical,? 

_10* Are^alJ of the options written ^o ^thay tfaey are not synonymous witl^ 
any other distxactc>r in the item? 

11, Is the correct response the one ot^ which competent critics would « 
agree? 

12, Are captions such as "none-of these," "all of these,*' etc., avoided? 

13, Are direct opposite pairs of options avoided? 

Are the distr^tors written so that they are not significantly 
different from the correct response with respect to; " 



14. Wording, phraseology? 

15. Granmiatical construction? 

16. Lengtb?^^ 
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Evaluation Information Chart, 62 
Evaluation -^lodel^i 35-51 
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Tylerian, 38-40 , 

CIPP, 40-44 
. CSE, 44-45 

formative evaluation, 46-^47 

summative evaluation, 47-48 



single subject design, 49-51 
Evaluation Question," 54 
Evaluation System, 53 

design .of, 53-77 
JEvaluation Techniques 

ac^eiveflient .test-s,'15, 

Crit'erion-referenced measures, 
16, 19-20 

maftery, 17 

no^m-refejrenced, 16 * ^ 

F-or.ati;e.(^Eva.„a;ioaiodel.) ' 

Guidance Program Evaluation, 86-87 

Hansen, Kenneth H., 4 

Harris, li?^0T L., 18 

Hemphill, John K*, 5 \ * 

Individual Reporting Fgrm," 60 v 

r 

Job Progress Evaluation, 88-91 

Mastery ( see Evaluation Techniques) 
Module^Evaluation Criteria, 63-68 
Moral-Ethical Considerations, 74-75 

Needs Assessment, 4 4 

Prograni^Evaluation Criteria,. 
6^ 63-68 

Program Evaluation Review Technique 

(PERT), ^9-73 , 
Program Plan, 4 * . 

Project Director Activity 

Evaluation Form, 61 
S ' ' , 

Safety Evaluation, 82-83 
Scales, 18 

Single Subject Design 

( see Evaluation Models) 
Stufflebeam, Daniel L., 40-44 

(see CIPP Model) 
Summative Evaluation 

( see Evaluation Models) * 

Table Qf Specif icE^tions , 55-58 ' 
. Ta^k Assignment Check Form, 59 
Taxohofhies , 20 * 
ana^lysis^of cognitive, 26 
^ concept of, 21 

definition of, tl-22 
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cojigruencyi 23 

decision makingi 24 « 

designing 'Appropriate test iteni&|^24 

domains, ^'24-25 ^ 

^ekrning problem diagnoses, 24 

learning to learp^ 23 
^ level of cognitive, 26 ^ 

range of obj'ectives, 23 * * , 

, reinforcing of learning, 23 ^ 

sequencing objectives, 23 

use of, 20 * ! , . 

vocational education, 23 
Test Development Instrument Guide, 92-94 
Tyler, RalpblW., 38*^^ . 

( s,ee also^ Evgluation Models) * * 

■ * ' 

Vocational Needs,* 4' ' • 
■ 

Youth. Organization Evaluation, o4-85 



